AI, Global Health and Algorithmic Society: Biodata’s democratic use and regulation

Abstract

Global health policies within an algorithmic society are developed through tensions between citizenship rights and market interests. In a “global health” timeline, the development of artificial intelligence (AI) has added greater complexity to these policies due to its potentialities and risks. If, on the one hand, the health sector is under the domination of the market’s techno-digital policies, which have undoubtedly brought improvements to the industry through the evolution of AI, on the other hand, one cannot ignore the abusive position of business corporations that feed their algorithmic networks through the mega data flows. In this context, two main points are analysed. The first highlights two ways of using data in the health sector that reflect a democratic and a non-democratic use of the same. While the democratic use emphasizes the potential of AI in the health sector, the non-democratic use reflects the tensions/risks arising from data extraction combined with financial and business logics, with an impact on privacy and security, and transparency. The second, seeking to eliminate or mitigate these main risks, analyzes the axis of governance and regulation of global health data in relation to AI.

Keywords:
Artificial Intelligence; Global Health; Algorithmic Society; Use of Biodata.

Resumo

As políticas globais de saúde no âmbito de uma sociedade algorítmica desenvolvem-se a partir de tensões entre direitos de cidadania e interesses do mercado. Numa cronologia da “saúde global”, o advento da inteligência artificial (IA) veio adicionar uma maior complexidade a essas políticas devido às suas potencialidades e riscos. Se, por um lado, o campo da saúde se encontra sob a dominação das políticas tecnodigitais do mercado, que indubitavelmente trouxeram melhorias ao setor através da evolução da IA, por outro lado, não se pode ignorar a posição abusiva das corporações empresariais que alimentam as suas redes algorítmicas por meio dos megafluxos de dados. Nesse contexto, analisam-se dois pontos principais. O primeiro assinala duas formas de utilização dos dados no setor da saúde que refletem um uso democrático e um uso não democrático dos mesmos. Enquanto o uso democrático enfatiza as potencialidades da IA no setor da saúde, o uso não democrático reflete as tensões/riscos que derivam do extrativismo dos dados aliados a lógicas financeiras e empresariais, com impacto na privacidade, na segurança e na transparência. O segundo, e procurando eliminar ou mitigar esses principais riscos, analisa o eixo da governação e regulação dos dados globais da saúde relativamente à IA.

Palavras-chave:
Inteligência Artificial; Saúde Global; Sociedade Algorítmica; Uso de Biodata.

Introduction

Risks have constantly threatened the physical and social integrity of peoples and the security of individuals, but they have acquired a different nature in the globalized and interconnected societies of late modernity in crisis (Reckwitz; Rosa, 2024), resulting from the productive forms of the risk society that create manufactured uncertainties (Beck, 2000; Giddens, 1991; Bauman, 2001). By exposing individuals to visible and invisible risks, this society generates a “culture of risk” and a culture of fear (Boucheron; Corey, 2015), underlying the feeling of ontological insecurity. In the field of Health, the paradoxes inherent in the risk society produce, simultaneously, effects of totalization and effects of individualization. Regarding the effects of totalization, we saw that, during the COVID-19 pandemic, anarcholiberalism (Foucault, 1979) calls on the State not to abdicate its role as a privileged actor in biopolitics. The effects of individualization were observed when individuals, faced with state deficits and the emphasis placed on personal and collective culpability, are held accountable for their health choices. In this sense, both effects exacerbated the security vs. freedom dilemma, showing that global health policies continue to hold individuals accountable for managing the risks to their existence, creating tensions between citizenship rights and market interests.

Because of these paradoxical equations, the last five decades have seen perverse effects emerge in the field of Health, particularly concerning the dismantling of public health institutions, whose functions are being delegated to private institutions (Lash, 2003, p. 13), revealing the decline of the welfare state, the result of neoliberal policies initiated in the 1980s (Rosanvallon, 1991). The “institutionalized individualism” emerged from this situation (Beck; Beck-Gernsheim, 2003), through which individuals renounce traditional public security mechanisms, their fundamental rights, that is, the resources offered by modernity. These resources, however, were implemented during the industrial modernity of the 19^th century, when infectious diseases became a significant public policy concern. Beginning in 1851, for example, some international conferences were held to establish health policies, thus initiating an international public health organization. In the early 20^th century, the International Office of Public Hygiene was created (1907) and, after the First World War, the Health Organization of the League of Nations, which was followed by the founding, in 1948, of the World Health Organization (WHO) (Guimier, 2022).

However, from the beginning, the implementation of public health protection devices came up against market interests, as shown by the example of quarantine measures proposed during periods of spread of infectious diseases, considered very costly for commerce (Kerouedan, 2013). During this century, the system acquired a multipolar character by integrating regional structures (European Union and others) and, equally, financial and commercial institutions such as the World Bank, the International Monetary Fund, and the World Trade Organization, which did not fail to cause obstacles (Guimier, 2022). On the other hand, this economic-financial vision of health on an international scale gave way to a more comprehensive conception of health, associated with poverty, which led to the implementation of developmental policies aimed at more impoverished countries, which proved insufficient. In the last two decades of the 20^th century, the impact of globalization on health led to a global program for this sector, which sought to evaluate global and local determinants, national programs, and the actions of multinational stakeholders (Kickbusch, 2024).

In the 1990s, a new parameter underpinned Public Health policies: the security parameter, which would be combined with the financial parameter. Priority was then given to factors such as the institutional weaknesses of more impoverished countries, population mobility, the growth of megacities, and other social variables that favor the spread of infectious diseases. In the United States, for example, the application of these criteria - economic security and national security - determines health policies: in 1996, the Clinton administration published a standard aimed at combating infectious diseases, due to their economic consequences; and the national security criterion appears in the field of Health before the 2001 terrorist attacks, notably in a 1997 report by the Institute of Medicine, which used the term “Global Health” for the first time to designate the health threat to the vital interests of the United States¹.

In reality, the notion of Global Health refers to a series of programs, institutional and operational actors, and global partnerships working to improve health in the Global South - that is, in countries in Asia, Africa, and Latin America. The 2000s saw an epistemological leap from the “health and development” paradigm to the “Global Health” paradigm, which, however, did not abandon this political strategy. However, the complexity of situations arising from globalization and political instability led to the realization that disease-fighting strategies should be considered collectively. Forms of geopolitical cooperation were created at the level of countries, international and national stakeholders, which promoted the circulation of health policies across different networks, including community and traditional medicine networks (Kerouedan, 2013).

In 2000, nearly two hundred countries established the Millennium Development Goals (MDGs) with the goal of achieving minimum global progress in reducing poverty, social inequalities, education, clean water, and access to healthcare by 2015. Between 2000 and 2007, significant global funding was allocated to this sector, originating from public-private partnerships linked to the pharmaceutical industry. One can see, then, that the global health concept corresponds not only to population and epidemiological criteria, but also to commercial interests and geopolitical situations (Kerouedan, 2013). These interconnections have given rise to global health governance, which involves, as mentioned, states, United Nations agencies, and the participation of private organizations. The global geopolitics of health has been constantly transformed by the political balances and imbalances inherent in the restructuring of capitalism².

We are facing the emergence of new globalization forms, among which is the one we will reflect on in this article: algorithmic globalization. The field of global health and the advent of artificial intelligence (AI) have added a greater complexity of opportunities and risks. Bioinformatics and AI have enabled a true revolution in opportunities by modifying scientific research procedures, enabling early detection of pathologies, improving diagnoses, ensuring higher-quality care, and so on. As for the risks, they refer not only to difficulties in the patient-doctor relationship, but above all to how data is used, accessed, and manipulated, raising questions of security and transparency in Health.

While the Health sector is dominated by the market’s techno-digital policies, which have undoubtedly brought improvements driven by the evolution of AI, we cannot ignore the abusive position of companies in the sector that feed their algorithmic networks through the mega flow of data. Therefore, concerns about data protection are growing in an increasingly digital context. In addition to personal health data, genetic and biometric data, known as “biodata”, become important information, posing an increased risk to individuals’ privacy. Through data extraction, the digital market and the use of AI enhance the use of unique information extracted from this type of data, challenging regulation within the ecosystem of healthcare tools and services. Furthermore, in an era in which logics of transparency correspond to a politics of visibility of the information regime, AI has transformed the form of biopolitical domination of industrial capitalism (Foucault, 1979) into a “biodata” domination, in which it is no longer bodies and energies that are exploited, but information and data. In the Health sector, the balance between transparency, privacy, and security becomes very hard to achieve due to the risk of sensitive personal data being exposed. Thus, in the evolution of risk society, we see the advent of an algorithmic society supported by a repertoire of data-driven technologies, which adds new layers to societal governance through unique modes of knowledge and creation (Peeters; Schuilenburg, 2021).

Thus, recognizing that the use of AI in healthcare encompasses several ethical and security issues (OECD, 2024), this article focuses on the use of data using AI in the Health sector. Aligned with the above, two forms of data use in the Health sector are highlighted, reflecting data democratic and undemocratic uses. While democratic use emphasizes the potential of AI in Health, undemocratic use reflects the tensions and risks to “fundamental rights” arising from data extraction combined with financial and corporate logic, with a negative impact on privacy, security, and transparency. The second approach, seeking to eliminate or mitigate the main risks, analyzes the governance and regulation of global health data vis-à-vis AI. Thus, it begins with a global/transnational perspective based on the WHO guidelines, and then analyzes existing EU regulations.

1. Health data in the age of AI: from democratic to undemocratic use

In a new global risk landscape, where the idea that algorithms are a solution to the most significant challenges of our time, particularly in Health, impressive amounts of data are generated about our daily actions, with algorithms processing and acting on this data to make decisions that generate, control, and stimulate behavior in everyday life (Peeters; Schuilenburg, 2021). This fact means that the use of algorithms not only expands the possibilities of current control and surveillance but also introduces a new paradigm characterized by greater rationality of governance and a change in the functioning of power, constituting a roadmap for action at the individual and collective levels (Peeters; Schuilenburg, 2021, pp. 5-6). In this sense, through its mode of action, AI accentuates a qualitative change through the use of algorithms, which has an increasing influence on sociopolitical and economic governance forms (Magalhães, 2023, p. 79). This new characteristic results in democratic and non-democratic data use forms. While democratic use represents progress towards the common good and respects fundamental rights, non-democratic use through the accumulation, manipulation, and abusive use of individual and collective data means that “risks can be used in a targeted manner as an instrument of legitimation” (Beck, 2013, p. 45)³.

The democratic use of AI with a positive impact improves the well-being of people and the population, bringing greater patient engagement and medication adherence, contributing to disease management, in an approach that reinforces the individualization and personalization of care (WHO, 2023).

This technology, which includes machine learning, natural language processing, and robotics, is applied to almost every field of medicine, potentially contributing to biomedical research, medical education, and healthcare delivery (Rigby, 2019). Among the positive aspects is its ability to integrate and learn from large clinical datasets, enabling AI to perform functions in diagnosis, clinical decision-making, and personalized medicine.

The application of AI in Health continues to accelerate rapidly, with potential applications across the sector, including drug discovery, virtual clinic appointments, disease diagnosis, prognosis, medication management, and health monitoring (Bajwa et al., 2021). AI helps solve some of the most pressing healthcare problems, such as cures for cancer and other serious diseases; improve access, speed, and accuracy of diagnosis; detect Public Health threats; prevent chronic diseases; and provide tools to strengthen the healthcare workforce (Anderson; Sutherland, 2024).

The assistance provided by AI technologies is mainly realized in machines that monitor the patient’s health conditions, in diagnostic systems, treatment recommendations, and digital health applications, favoring a model centered on the patient and their particular characteristics through P4 Medicine (predictive, preventive, personalized, and participatory) (Freitas, 2023). AI technologies in Health allow for the analysis of symptoms and tests, offering suggested diagnostic hypotheses and, consequently, treatment options. They also help reduce “blind spots” in clinical reasoning, offering additional support so that doctors can explore all possibilities more confidently (Lapa, 2024).

It can also contribute to better healthcare management, particularly in administrative tasks that consume a large portion of doctors’/nurses’ time. Innovations brought about by AI allow healthcare professionals to devote more time to interacting with patients and less time to bureaucratic tasks (Lapa, 2024). Overall, AI can have a positive impact on clinical practice, biomedical research, public health, and healthcare administration (Lekadir et al., 2022).

However, AI is used to make decisions that may affect the health and the physical and emotional well-being of citizens. Also, in most cases, it lacks the use of personal data, which contains sensitive information about patients (Cansado, 2024). Managing this balance is particularly challenging in healthcare, where there are significant opportunities and profound risks, which leads us to analyze the undemocratic use of AI in Health.

The fundamental question lies in how personal and collective data are accumulated, manipulated, and used, based on “algorithmic extractivism” (or “data extractivism”) (Crawford, 2021). This concept is closely linked to the “generalized idea that only through access to data can relevant patterns be found and knowledge evolution achieved” (CNEV, 2024, p. 26).

This concept establishes an analogy between information management and the data extraction mining industry, where digital data is a raw material that can be extracted, commercialized, refined, and processed through platforms that enable a new accumulation regime (Bueno; Schultz, 2021). This is also why this process has also been called “data colonialism”, a global process of extraction in which, in the new digital age, what is now appropriated is human life through its conversion into data (Couldry; Mejias, 2019, p. xix). This means that the appropriation of human life data functions in a way that is interconnected with social arrangements and technological infrastructures, some that emerged during the previous capitalism and others new, which allow this data to be transformed into a commodity. The dizzying acceleration of AI, supported by transnational megacorporations, infrastructures, and platforms, generates an economic order based on human experience as free raw material to fuel the production of goods and services (Zuboff, 2019).

In this sense, AI innovation is constantly restructuring the global economic, political, and institutional arena, forcing States to adapt their policies to new global objectives and strategies; otherwise, they will not be sufficiently competitive in the global market. This new “global digital” backdrop significantly affects states’ ability to compete, leading to excessive competition in which governments and companies perpetuate inequalities, both locally and globally (Tworek, 2022). It is under this extractive logic that supports the undemocratic use of data that we have chosen to analyze two main risks related to the application of AI in healthcare⁴.

The first challenge lies in privacy and security issues, related to sharing personal data without fully informed consent, the reuse of data without the patient’s knowledge, data breaches that may expose sensitive or personal information, and the risk of harm - or even potentially fatal harm - from cyberattacks on AI solutions, at the individual and hospital or health system level (Lekadir et al., 2022, p. I-II). The effectiveness of consent to data use becomes crucial, particularly for the development of AI systems. Since medical data may be shared with third parties, informed and valid consent from citizens for the collection of their personal data is essential for the development of other AI systems. The sizeable volume of information and quality data increases the risk of privacy breaches (CNEV, 2024, p. 55). Furthermore, the fact that most people have skills in using technology but little knowledge of how it works masks a “trust” in the process that may not correspond to reality, which increases the risks regarding data availability and privacy.

However, the fact that most AI systems and the data they collect are associated with large technology corporations (Freitas, 2023) weighs further on this situation. This situation creates an imbalance between public and private institutions and raises concerns about these corporations’ privacy violations and the misuse of personal data (Cansado, 2024, p. 8). The data collected is increasingly coveted by the marketing industry, creating serious conflicts of interest, for example, between private companies that produce or process the data and aim to make a profit, and the State, which is responsible for its use in the best interests of the individuals whose data were collected, and of society in general (CNEV, 2024, p. 49). This fact illustrates how cybersecurity and data quality are highly relevant, since in the provision of healthcare using AI systems and applications, several circumstances can arise, such as the access, availability, and transferability of health data that can raise privacy concerns (CNEV, 2024, pp. 48-50). There is an imbalance between those who acquire, accumulate, analyze, and control the data and those who provide the data and have little control over its use.

The second is related to privacy and security and is the need for transparency required for the use of AI, to ensure the reliable verifiability of processes and results (CNEV, 2024, p. 27). Transparency is the ethical principle most frequently found in general guidelines for the use of AI and is a primary principle for AI in Health, meaning that systems must be intelligible and explainable to health professionals, patients, users, and regulators, per the understanding capacity of each group and even each individual (Dourado; Aith, 2022, p. 3). Transparency is thus closely linked to the traceability and explainability concepts, reflected at two distinct levels: 1) Transparency of AI development and use processes (traceability); and 2) Transparency of AI decisions themselves (explainability) (Lekadir et al., 2022, p. I-II).

Regarding the first, the issue of transparency and explainability refers to identifying the sources of AI errors and defining who and/or what is responsible for them (Lekadir et al., 2022, pp. I-II). This issue is directly related to “algorithmic responsibility” (accountability) as a crucial factor for trustworthy and applicable AI in Health. However, legal gaps remain in current national and international regulations regarding liability for errors or failures in AI systems, especially in medical AI. It is challenging to define roles and responsibilities due to the multiple stakeholders involved in the medical AI process, from design to implementation (e.g., healthcare professionals or AI developers) (Lekadir et al., 2022, pp. I-III). Notably, the lack of explainability of AI systems can also pose difficulties for doctors in understanding, for example, the machine’s diagnosis (Cansado, 2024).

Regarding the second level, the right to explanation regarding automated decisions is fundamental to algorithmic transparency, and is a fundamental element in algorithm regulation (Dourado; Aith, 2022). Explainability can combat events such as black-boxing, where the process fails to clarify how input data translates into the final result. This is especially important when addressing systemic human biases that are often embedded in AI models, including pervasive and deep-rooted biases based on sex and gender, race and ethnicity, age, socioeconomic status, geographic location, and urban or rural contexts. The most common causes of AI bias in Health are biased and imbalanced datasets that may be based on structural preconception and discrimination (systemic discrimination that is embedded in how data are collected or how doctors treat their patients), and disparities in access to quality equipment and digital technologies, as well as a lack of diversity and interdisciplinarity in technology, scientific, clinical, and policymaking teams (Lekadir et al., 2022, p. I-II).

In Health, there is also the fact that medical records are not adapted “to the specific analysis functions of AI, with the records being generated solely for treatment and billing purposes, leading to a more significant increase in systemic biases” (Duffourc; Giovanniello apudCansado, 2024, p. 8). However, it is important to note that excessive transparency can also affect privacy, in the sense that an adequate and thorough explanation of the process may imply the exposure of sensitive and confidential personal data (CNEV, 2024, p. 27), especially in an era of control through “biodata”, as mentioned above.

2. Global governance and regulation of AI in Health

The risks identified in the previous topic are made even more serious by the lack, or insufficiency, of regulation, since there are no universally applicable AI guidelines (or even principles), much less any that apply to the Health sector in particular.

In its report “Regulatory considerations on artificial intelligence for health” (2023), the WHO emphasizes the importance of establishing the safety and effectiveness of AI systems so they are appropriately available to those in need⁵. With clearer regulations, the WHO hopes the new technology will improve medical diagnosis and complement the knowledge and skills of health professionals.

In this regard, the WHO advocates for the creation of robust legal and regulatory frameworks to protect patient privacy, security, and integrity. In response to countries’ growing need to responsibly manage the rapid rise of AI-based health technologies, the publication outlines six areas for regulation. The areas are: transparent documentation of the product life cycle from development; risk management; external data validation; data quality; privacy protection; and collaboration among all stakeholders, including patients and health professionals (WHO, 2023, p. xii-xiii). The WHO emphasizes that AI systems are complex and depend not only on the code they are built with, but also on the data they are trained on, which derives from clinical settings and user interactions, for example. The analysis aims to outline key principles that governments and regulatory authorities should follow when developing new or adapting existing guidance on AI at the national or regional level⁶.

More recently, in its quest for global governance for the future of AI in healthcare, the WHO released the report “Ethics and governance of artificial intelligence for health - Guidance on large multimodal models” (2024). The report addresses generative AI technology, the development of which is changing approaches to healthcare. It can process several types of requests, such as text, videos, and images, to generate results and mimic human communication.

The new WHO guidelines reflect the importance of directing and regulating the use of these models (such as ChatGPT) to ensure ethical and practical benefits for global health, outlining a set of recommendations for governments, technology companies, and healthcare providers. The agency recommends that governments define standards for the development and deployment of large multimodal models (LMMs) in healthcare. The proposal suggests the creation of a regulatory agency to evaluate and approve LMMs, in addition to the implementation of post-launch audits and impact assessments conducted by independent third parties. The WHO also highlights the importance of involving potential users and stakeholders, such as doctors, researchers, and patients, from the early stages of AI development, in a structured, inclusive, and transparent process⁷.

2.1 European regulation of AI in healthcare: GDPR and the AI Act

In the European Union (EU), existing legislation regarding data protection and security came into force in 2018 with the General Data Protection Regulation (GDPR)⁸, which “protects individuals whenever their data are processed by the private sector and most of the public sector”⁹.

The GDPR is part of the EU’s data protection reform, along with the Data Protection in Law Enforcement Directive and Regulation (EU) 2018/1725 on the processing of personal data by the EU institutions, bodies, offices, and agencies (idem). The GDPR sets out individual rights and rules applicable to companies. In short, regarding individual rights, the GDPR facilitated citizens’ access to their data, providing information on how data is processed and ensuring that this information is made available clearly and understandably; it facilitated the transmission of personal data between service providers by establishing a new right to data portability; it establishes the “right to be forgotten”, asserting the right to erasure of data whenever there are no legitimate reasons for retaining it; it also establishes the right to know when personal data has been breached, with companies and organizations having a duty to notify the competent data protection supervisory authority and, in cases of severe data breaches, also the affected individuals. For businesses, the GDPR creates a level playing field for all companies operating in the EU internal market.

In turn, in the Health sector, data classified as special by the GDPR (Article 9, N°1) is analyzed. As data subjects, patients have the right to know how, and to what extent, the information collected will be used (Articles 15, 16, and 18). These rights are relevant to ensuring the effectiveness of consent to data use, which is necessary, in particular, for the development of AI systems (Cansado, 2024, p. 8). Although it does not explicitly identify AI, Article 22(1) of the GDPR provides that “The data subject shall have the right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her”. According to Cansado’s analysis (2024, p. 9), this includes decision-making based exclusively on AI systems. The same applies to the requirement for algorithmic transparency, which in the healthcare field, according to the GDPR, is embodied in the right to informed consent, for performing medical procedures and collecting personal data (Articles 6a and 7), and also in liability when there is harm caused by the AI system (Cansado, 2024, p. 6).

Along with the GDPR, we should underscore the “Proposal for a Regulation of the European Parliament and of the Council on the European Health Data Space”¹⁰, which was approved by the European Parliament in April 2024 and aims to facilitate citizens’ access to their electronic health data in all member countries, while simultaneously promoting interoperability between European health systems. Building on the GDPR, and addressing “the uneven application and interpretation of the GDPR by Member States [which] creates considerable legal uncertainty, resulting in obstacles to the secondary use of electronic health data”, it establishes the European Health Data Space (EHSD), ensuring that individuals in the EU have, in practice, greater control over their electronic health data¹¹.

It should be noted that currently, the regulations applicable to AI “medical tools” in the EU are the Medical Device Regulation (MDR) 2017/745 and the In Vitro Diagnostic Medical Device Regulation (IVDR) 2017/746, which were approved in 2017. However, because they were created at a time when AI was still in an early stage of its development, many specific aspects of AI are not considered, such as the continuous learning of AI models or the identification of algorithmic biases (Lekadir et al., 2022).

Therefore, regarding AI, and although it is of general scope, not considering the specificities and risks of AI in Health (unlike the MDR and IVDR regulations), the AI Act approved in June 2024 and which will be implemented in phases, with completion expected in 2027, is an instrument of great legal relevance¹² for the framework of the uses of AI in Health.

The AI Act is the first comprehensive legal framework for AI worldwide, with the primary objective of promoting trustworthy AI within and outside Europe, ensuring that AI systems respect fundamental rights, safety, and ethical principles. To this end, it follows a four-level AI risk analysis methodology: 1) unacceptable risk; 2) high risk; 3) low risk; and 4) minimal risk. The regulation is complex and robust, with requirements for each level based on the identified risk. Therefore, we briefly outline its main principles and standards here:

Address the risks specifically created by AI applications; prohibit AI practices that present unacceptable risks; determine a list of high-risk applications; establish precise requirements for AI systems for high-risk applications; define specific obligations for deployers and providers of high-risk AI applications; require a conformity assessment before a given AI system is put into service or placed on the market; apply enforcement measures after a given AI system has been placed on the market; establish a governance structure at European and national levels.¹³

Prohibited uses of AI systems can be identified in Health, namely those listed in Article 5, Paragraph 1, Subparagraph a), which refers to “the placing on the market, putting into service or use of an AI system that deploys subliminal techniques beyond a person’s consciousness or purposefully manipulative or deceptive techniques, with the objective, or the effect of materially distorting the behavior of a person or a group of persons by appreciably impairing their ability to make an informed decision, thereby causing them to take a decision that they would not have otherwise taken (…)”.

In medical diagnoses that use AI systems, it is essential to find a balance between an informative result for the user and even for the doctor himself, without this result being biased and conditioning their choice, to guarantee the effectiveness of informed consent (Cansado, 2024, p. 11). Along the same lines, subparagraph “g” of Article 5, N°1, prohibits the use of “biometric categorization systems that individually classify individuals based on their biometric data to deduce or infer their race, political opinions, trade union membership, religious or philosophical beliefs, sex life, or sexual orientation” (AI Act, 2024). With regard to high-risk systems, and according to Article 6, N°1, these are divided into two criteria:

(a) The AI system is intended to be used as a safety component of a product, or the AI system is itself a product, covered by the Union harmonization legislation listed in Annex I;

(b) The product whose safety component pursuant to point (a) is the AI system, or the AI system itself as a product, is required to undergo a third-party conformity assessment, with a view to the placing on the market or the putting into service of that product pursuant to the Union harmonization legislation listed in Annex I (AI Act, 2024).

Now, considering subparagraph “a”, AI systems used in the medical sector would mostly be considered high-risk systems. This is evidenced by the fact that Annex I (points 11 and 12) explicitly addresses medical devices, including devices for in vitro diagnostics. Regarding subparagraph “b”, and according to Annex III, paragraph 1, this includes:

a) Remote biometric identification systems (…); b) AI systems intended to be used for biometric categorization, according to sensitive or protected attributes or characteristics based on the inference of those attributes or characteristics; c) AI systems intended to be used for emotion recognition (AI Act, 2024).

As Cansado (2024, p. 11-12) points out, it is important to note that one of the uses considered to be high risk is the use of systems by public authorities to assess eligibility for assistance services, particularly in the health sector (Annex III, N° 5, a), as well as the patient triage system in hospital emergency rooms (Annex III, N° 5, d). Notably, there is a need for human oversight to mitigate risks to the health, safety, and fundamental rights of citizens, especially in high-risk systems such as diagnostic systems (Article 14, N° 2). This is justified because AI systems can be subject to bias, either due to the data input or how the system was programmed. Therefore, the AI Act, in its Article 14, N°3, subparagraph “a”, requires producers of AI systems to incorporate human oversight measures into the system itself, or to identify such measures, prior to placing the product on the market (Cansado, 2024, pp. 14-16).

The AI Act introduces important quality and safety requirements for AI used in the healthcare sector, especially regarding data governance for AI medical devices. However, it also has some limitations, namely its horizontal nature and lack of sectoral interpretation (Kolfschooten; Oirschot, 2024). These include: 1) The lack of comprehensive oversight and accountability mechanisms for low-risk health-related AI systems, which may result in the widespread emergence of ineffective, unproven, and potentially harmful AI systems; 2) The process and purpose of fundamental rights impact assessment are ambiguous, as they require listing potential impacts on rights but contain no clear obligation to assess their acceptability; 3) The lack of clear and mandatory fundamental rights assessments for private healthcare providers may result in disparities in the protection of patients’ rights between providers and between Member States; 4) If health-related AI systems are used for national security purposes (e.g., screening for biological threats), they are exempt from all rules; and 5) The scientific research exemption provided for in the AI Act potentially leaves room for evasion of the regime for AI used in medical research and clinical trials, which may pose a risk to patients (Kolfschooten; Oirschot, 2024, p. 3-4).

Conclusion

In this article, we considered algorithmic society as a development of risk society within the context of global health policies. AI adds layers of complexity to the healthcare sector, particularly regarding how data is used. In an algorithmic society where a massive amount of data, used through algorithms, manages individual and collective behavior, impacting political and economic governance, one can identify how this translates into democratic and undemocratic forms of data use in Health. While democratic use represents progress toward the common good and respects fundamental rights, undemocratic use highlights how “algorithmic extractivism”, through the accumulation, manipulation, and abusive use of individual and collective data, results in increased risks for citizens.

From our analysis, and regarding undemocratic uses in the Health sector, we first underscore the issue of data privacy and security and, second, the issue of transparency. The power of private companies that hold and control healthcare data translates into new individual and collective insecurity forms that are simultaneously enhanced and camouflaged by AI technologies.

This fact leads to the pressing need for AI regulation in healthcare. Notably, at the global level, are the WHO’s main guidelines on the use of AI in the healthcare sector, as well as, at the EU level, current legislation that seeks to eliminate or mitigate the risks mentioned above. At the EU level, and seeking to consolidate its position as a “normative and legislative vanguard” when it comes to AI regulation, there is substantial concern about the risks of its use. Even so, we can conclude that the existing regulation is mainly of a general nature. Although it fulfills the functions of protecting and securing individuals and their data, there is still a lack of sectoral regulation that corresponds to the growing challenges posed by the use of AI in healthcare.

Thus, we conclude with three ideas. The first is that AI technologies have a political, economic, and social impact due to their raw material (data), reflected in how the business and financial sectors instrumentalize them. This factor leads us to the second idea: that in a global backdrop, and specifically in the healthcare sector, where data assumes particular relevance due to the type of information it contains, robust, effective, and accessible regulation is urgently needed; which leads to the third idea: a regulatory framework for AI in healthcare (and also at a general level) must be jointly produced in a concerted and democratic manner, with the participation of several stakeholders, addressing the different direct and collateral effects of its use and, in this sense, reinforcing humanity in decision-making and closing the gaps in the manipulation of its risks.

• 1
  This was also the understanding of the National Intelligence Services Commission (1999-2008), which, in its reports, considered the disease a “non-traditional threat agent” to the country’s security (Kerouedan, 2013).
• 2
  Since 2017, China has launched a national development plan in this sector, aiming to generate approximately US$60 billion by 2025 (Callegarin; Callier, 2021). More recently, emerging economies in Brazil, China, and India have also intervened in global health governance, a topic discussed at the G8, G20, and United Nations summits.
• 3
  In an analysis of the 2008 financial crisis, the author states that it is at this point that a relationship is established between the theory of risk society and the reflections of Carl Schmitt, given that risk society becomes a “(latent) revolutionary society, in which it is no longer possible to clearly distinguish between a normal state and a state of exception” (Beck, 2013, p. 46).
• 4
  Notably, a study by the European Parliament’s Panel for the Future of Science and Technology affirms that there are seven main risks of AI in medicine and healthcare: 1) Patient harm due to AI errors; 2) The misuse of medical AI tools; 3) AI bias and the perpetuation of existing inequalities; 4) Lack of transparency; 5) Privacy and security issues; 6) Accountability gaps; and 7) Obstacles to implementation (Lekadir et al., 2022). However, for this article, we focused our analysis on data use, specifically, issues related to privacy, security, and transparency.
• 5
  In 2021, the WHO also published comprehensive guidance on the ethics and governance of AI for health (WHO, 2021). The WHO consulted 20 leading AI experts, who identified both the potential benefits and potential risks of using AI in healthcare and issued six consensus-based principles for consideration in the policies and practices of governments, developers, and providers using AI. The principles should guide the development and implementation of AI in healthcare by a wide range of stakeholders, including governments, public sector agencies, researchers, businesses, and implementers. The principles are: (1) protecting autonomy; (2) promoting human well-being, human security, and the public interest; (3) ensuring transparency, explainability, and intelligibility; (4) fostering accountability; (5) ensuring inclusion and equity; and (6) promoting AI that is responsive and sustainable.
• 6
  Cf: https://news.un.org/pt/story/2023/10/1822142
• 7
  Cf. https://news.un.org/pt/story/2024/01/1826467
• 8
  Available at: https://eur-lex.europa.eu/legal-content/PT/TXT/?uri=CELEX:32016R0679
• 9
  Cf. https://eur-lex.europa.eu/PT/legal-content/summary/general-data-protection-regulation-gdpr.html
• 10
  Available at: https://eur-lex.europa.eu/resource.html?uri=cellar:dbfd8974-cb79-11ec-b6f4-01aa75ed71a1.0010.02/DOC_1&format=PDF
• 11
  As mentioned, the European Health Data Space (EHDS) is based on legislation such as the GDPR, as well as Regulation (EU) 2017/745 on medical devices (Medical Devices Regulation) and Regulation (EU) 2017/746 on in vitro diagnostic medical devices (In Vitro Diagnostic Devices Regulation), the proposed Artificial Intelligence Act (AI Act), the proposed Data Governance Act and the proposed Data Act, Directive (EU) 2016/1148 on network and information security (NIS Directive) and the Cross-border Healthcare Directive.
• 12
  Available at: https://eur-lex.europa.eu/legal-content/PT/TXT/PDF/?uri=OJ:L_202401689
• 13
  Cf. https://digital-strategy.ec.europa.eu/pt/policies/regulatory-framework-ai

Data Availability Statement

The research data are available within the main text of the article.

References

• ANDERSON, B.; SUTHERLAND, E. Collective action for responsible AI in health. OECD Artificial Intelligence Papers, Paris, n. 10, 2024. DOI: https://doi.org/10.1787/f2050177-en
  » https://doi.org/10.1787/f2050177-en
• BAJWA, J. et al. Artificial intelligence in healthcare: transforming the practice of medicine. Future healthcare journal, v. 8, n. 2, p. e188-e194, 2021. DOI: https://doi.org/10.7861/fhj.2021-0095
  » https://doi.org/10.7861/fhj.2021-0095
• BAUMAN, Z. A modernidade líquida Rio de Janeiro: Zahae, 2001.
• BECK, U. A Europa Alemã - De Maquiavel a «Merkievel»: estratégias de poder na crise do Euro. Lisboa: Edições 70, 2013.
• BECK, U. La société du risque Sur la voie d’une autre modernité? Paris: Alto/Aubier, 2001.
• BECK, U.; BECK-GERNSHEIM, E. La individualización: El individualismo institucionalizado y sus consecuencias sociales y políticas. Barcelona: Paidós Ibérica, 2003.
• BOUCHERON, P.; COREY, R. L’exercice de la peur: usages politiques d’une émotion. Lyon: Presses universitaires de Lyon, 2015.
• BUENO, C.; SCHULTZ, M. Extractivismo de datos. Imaginación maquínica, 2021. Available from: http://imaginacionmaquinica.cl/extractivismo-de-dato Access in: 29 set. 2023.
  » http://imaginacionmaquinica.cl/extractivismo-de-dato
• CALLEGARIN, D.; CALLIER, P. Enjeux du déploiement de l’intelligence artificielle en santé. Actualités Pharmaceutiques, v. 60, n. 611, p. 21-24, 2021. DOI: https://doi.org/10.1016/j.actpha.2021.10.006
  » https://doi.org/10.1016/j.actpha.2021.10.006
• CANSADO, M. DA P. Inteligência artificial no setor da saúde: desafios jurídicos e regulação. GEE - Gabinete de Estratégia e Estudos do Ministério da Economia, 187. Lisboa: GEE, 2024. Available from: https://www.gee.gov.pt/RePEc/WorkingPapers/GEE_PAPERS_187.pdf Access in: 28 jan. 2025.
  » https://www.gee.gov.pt/RePEc/WorkingPapers/GEE_PAPERS_187.pdf
• CNEV - Conselho Nacional de Ética para as Ciências da Vida. Inteligência artificial (IA): inquietações sociais, propostas éticas e orientações políticas - Livro Branco. Lisboa: CNEV, 2024.
• COULDRY, N.; MEJIAS, U. The costs of connection: How data is colonizing human life and appropriating it for capitalism. Standford: Standford University Press, 2019.
• CRAWFORD, K. Atlas of AI: Power, politics, and the planetary costs of artificial intelligence. London: Yale University Press, 2021.
• DOURADO, D.; AITH, F. A regulação da inteligência artificial na saúde no Brasil começa com a Lei Geral de Proteção de Dados Pessoais. Revista de Saúde Pública, São Paulo, v. 56, p. 80, 2022. DOI: https://doi.org/10.11606/s1518-8787.2022056004461
  » https://doi.org/10.11606/s1518-8787.2022056004461
• FOUCAULT, M. Naissance de la biopolitique Cours au Collège de France 1978-1979. Paris: Gallimard/Seuil, 1979.
• FREITAS, A. Data driven approaches in healthcare: Challenges and emerging trends. In: ANTUNES, H. S. et al.). Multidisciplinary perspectives on artificial intelligence and the law. Springer, 2023. p. 65-80.
• GIDDENS, A. As consequências da Modernidade São Paulo: UNESP, 1991.
• GUIMIER, L. Géopolitique de la santé. La santé publique à l’épreuve des idéologies Paris : Éd. Le Cavalier bleu, Coll. Géopolitique, 2022.
• KEROUEDAN, D. Comment la santé est devenue un enjeu géopolitique. Le Monde Diplomatique, Juillet 2013, p. 16 -17.
• KICKBUSCH, I. Ilona Kickbusch highlights need for action to address the digital determinants of health. Digital Transformations for Health Lab, 4 Dec. 2024. Available from: https://dthlab.org/ilona-kickbusch-highlights-need-for-action-to-address-the-digital-determinants-of-health Access in: 28 jan. 2025.
  » https://dthlab.org/ilona-kickbusch-highlights-need-for-action-to-address-the-digital-determinants-of-health
• KOLFSCHOOTEN, H.; OIRSCHOT, J. The EU Artificial Intelligence Act (2024): Implications for healthcare. Health Policy, v. 149, p. 105152, 2024. DOI: https://doi.org/10.1016/j.healthpol.2024.105152
  » https://doi.org/10.1016/j.healthpol.2024.105152
• LAPA, E. O uso da Inteligência Artificial na medicina: transformação e desafios. Exame online, 12 out. 2024. Available from: https://exame.com/colunistas/opiniao/o-uso-da-inteligencia-artificial-na-medicina-transformacao-e-desafios/ Access in: 28 jan. 2025.
  » https://exame.com/colunistas/opiniao/o-uso-da-inteligencia-artificial-na-medicina-transformacao-e-desafios/
• LASH, S. Prefacio: Individualización a la manera no lineal. In: BECK, U.; BECK-GERNSHEIM, E. La individualización: El individualismo institucionalizado y sus consecuencias sociales y políticas. Barcelona: Paidós Ibérica, 2003.
• LEKADIR, K. et al. Artificial intelligence in healthcare: Applications, risks, and ethical and societal impacts. Brussels: European Parliament, 2022.
• MAGALHÃES, J. Democracia ou algoritmocracia? Ensaio sobre a defesa dos direitos humanos na era digital. Tictank, 20 maio 2023. Available from: https://tictank.pt/2023/05/20/democracia-ou-algoritmocracia/ Access in: 29 set. 2023.
  » https://tictank.pt/2023/05/20/democracia-ou-algoritmocracia/
• OCDE. AI in health: huge potential, huge risks. Paris: OCDE Publishing, 2024.
• PEETERS, R.; SCHUILENBURG, M. The algorithmic society: An introduction. In: SCHUILENBURG, M.; PEETERS, R. The algorithmic society technology, power, and knowledge London: Routledge, 2021. p. 1-15.
• RECKWITZ, A.; ROSA, H. La modernité tardive en crise Paris: Editions de la Maison des Sciences de l’Homme, 2024.
• RIGBY, M. Ethical dimensions of using artificial intelligence in health care. AMA Journal of Ethics, v. 21, n. 2, p.121-124, 2019.
• ROSANVALLON, P. La crise de l’Etat Providence Paris : Seuil, 1981.
• TWOREK, H. The path dependency of infrastructure: A commonly neglected aspect of platform governance. In: Centre for International Governance Innovation. Report Global Cooperation on Digital Governance and the Geoeconomics of New Technologies in a Multi-polar World Waterloo: Centre for International Governance Innovation, 2022.
• UNIÃO Europeia. AI Act (2024) Regulamento (UE) 2024/1689 do Parlamento Europeu e do Conselho, de 13 de junho de 2024 Cria regras harmonizadas em matéria de inteligência artificial e que altera os Regulamentos (CE) n.° 300/2008, (UE) n.° 167/2013, (UE) n.° 168/2013, (UE) 2018/858, (UE) 2018/1139 e (UE) 2019/2144 e as Diretivas 2014/90/UE, (UE) 2016/797 e (UE) 2020/1828 (Regulamento da Inteligência Artificial) (Texto relevante para efeitos do EEE). Bruxelas: Jornal Oficial da União Europeia, 2024.
• WHO. Ethics and governance of artificial intelligence for health: Guidance on large multi-modal models. Geneva: WHO, 2024.
• WHO. Ethics and governance of artificial intelligence for health. Geneva: WHO, 2021.
• WHO. Regulatory considerations on artificial intelligence for health Geneva: WHO, 2023.
• ZUBOFF, S. The age of surveillance capitalism: The fight for a human future at the new frontier of power. Great Britain: Profile Books, 2019.

• Editors:
  Carinne Magnago, Aurea Ianni

Publication Dates

• Publication in this collection
  15 Dec 2025
• Date of issue
  2025

History

• Received
  03 Feb 2025
• Accepted
  28 May 2025