How to govern health AI without blocking innovation? Hannah van Kolfschooten, researcher at the Centre for Life Sciences Law at the University of Basel, explains why innovation and regulation can go hand in hand and why AI chatbots have become 'shadow health systems'.
You serve on the World Health Organization's Technical Advisory Group on AI for Health. Could you give us some insight into the group's work?
"The WHO Regional Office for Europe established the Technical Advisory Group on AI for Health to advise on how AI can be used in healthcare in a way that is safe, ethical, equitable, and effective. The group brings together ten experts from disciplines including medicine, public health, computer science, ethics, and law."
"Our role is to advise WHO/Europe on integrating ethics, governance, regulation, and oversight into AI strategies. We also contribute to policy recommendations and capacity-building, and advise on how AI should be implemented, monitored, and evaluated to ensure it is safe, effective, transparent, accountable, respects human rights, and ultimately improves health outcomes."
"My own work within the group focuses on the legal and ethical dimensions of AI governance, particularly the protection of patients' rights and the development of effective regulatory frameworks."
When it comes to regulating AI in healthcare, where do you place yourself on the spectrum between innovation and precaution? Would you describe your approach as more liberal or more conservative?
"One of the recurring themes in international discussions is that innovation and precaution are often presented as opposites, whereas in healthcare they are closely connected. In healthcare, meaningful innovation depends on trust, and trust depends on robust safeguards. My approach to regulation is therefore evidence-based and proportionate: where AI has been shown to improve patient outcomes, we should enable and encourage its adoption."
"At the same time, we need appropriate safeguards, because in healthcare mistakes can have serious consequences for patients. Good regulation should not stand in the way of innovation, but help ensure that the AI is safe and delivers real clinical value."
In one of the papers you co-authored, you describe AI chatbots as the "emergence of shadow health systems." What do you mean by that?
"In the paper you mean, we argue that AI chatbots are increasingly performing functions that were traditionally part of the healthcare system, such as providing health information, mental health support, symptom assessment, and lifestyle advice."
"We describe this phenomenon as the emergence of "shadow health systems." The idea is that these technologies are beginning to influence people's health decisions without being fully integrated into formal healthcare. Unlike healthcare providers, many AI chatbots operate outside established clinical pathways and are not subject to the same standards of oversight, accountability, or quality assurance. Yet millions of people already rely on them when making decisions about their health."
One can argue that people have long relied on sources of health information such as books, websites, and Google searches, none of which have been subject to specific regulation. Why do you believe AI chatbots should be regulated differently?
"The key difference is not that AI chatbots provide health information, but that they increasingly perform functions that were traditionally carried out within the healthcare system. A book or a website provides static information, while AI chatbots engage in personalized conversations, interpret symptoms, influence care-seeking decisions, and, in some cases, may even substitute for contact with healthcare professionals."
"At scale, AI chatbots can influence when people seek care, how they interpret symptoms, and whom they trust. We argue that, where AI begins to influence healthcare in ways that affect individual health, it should be subject to proportionate safeguards that reflect that role."
How can we effectively regulate generative AI when its outputs are inherently unpredictable and even its creators cannot fully control its outputs? It can help save a life by correctly identifying a serious condition, but it also may provide dangerously misleading advice.
"What we're increasingly seeing across countries is a shift away from trying to regulate individual AI outputs towards governing AI throughout its lifecycle. That means robust testing before deployment, clearly defining the contexts in which a system may be used, continuously monitoring its real-world performance, and ensuring that serious incidents are reported and addressed."
"When generative AI is used in higher-risk settings such as healthcare, stronger safeguards are needed, including clinical validation, human oversight, clear escalation pathways, and accountability when things go wrong. The goal should not be to eliminate all unpredictability, but to ensure that the risk of uncertainty is not only borne by users seeking health advice."
China is introducing regulations aimed at limiting 'emotional dependence on AI'. Should Europe consider taking a similar approach?
"The EU is also taking action to limit emotional dependence on AI. The AI Act prohibits certain AI systems that manipulate people or exploit their vulnerabilities in ways that are likely to cause harm. The proposed Digital Fairness Act is also expected to strengthen the rules on addictive and manipulative digital design."
But is the EU AI Act sufficient to protect society from the risks posed by artificial intelligence?
"The EU AI Act provides a strong framework for managing AI risks for developers and deployers, particularly in high-risk sectors such as healthcare. At the same time, it does not provide detailed guidance on how healthcare professionals should use AI in clinical practice."
"Questions such as when clinicians should rely on AI, when they should challenge its recommendations, and how AI-supported decisions should be communicated to patients will need to be addressed through clinical guidelines, professional standards, and experience in practice. Whether the AI Act succeeds will ultimately depend not only on the legislation itself, but also on how it is implemented and continues to evolve as the technology develops."
AI has been used safely in healthcare for years – for example, in radiology – even before the EU AI Act. Is the concern about new AI-related risks sometimes overstated?
"I don't think the concerns are overstated, but neither should we overlook the many successes of AI in healthcare. Radiology is actually a good example of why both innovation and regulation are needed. AI has already improved workflows and diagnostic performance in many settings, but we've also seen important risks. For example, some breast cancer screening algorithms perform less well for women from minority backgrounds or women with dense breast tissue, leading to differences in false-positive rates."
"We've also seen challenges around explainability, accountability, and how clinicians should use AI in practice. One of the biggest shifts we're seeing internationally is that the conversation has moved beyond whether AI works technically to whether it actually improves patient outcomes in routine clinical practice."
Already 65% of physicians in the United States use OpenEvidence, praising its ability to support clinical decision-making. The company has withdrawn from the European market, citing the EU AI Act. Does restrictive regulation risk driving away valuable innovations alongside the harmful ones?
"It's certainly important that regulation doesn't create unnecessary barriers to innovation. But I don't think we should conclude from one example that Europe is becoming hostile to clinical AI. We're seeing a growing number of European alternatives emerge, designed around European healthcare systems, national clinical guidelines, and local regulatory requirements."
That's important because a tool designed for the US healthcare system cannot simply be transplanted into Europe, where healthcare is organized considerably differently.
We are also seeing the rise of 'shadow AI'. Even when healthcare organizations prohibit the use of AI tools, many clinicians continue to use them unofficially.
"One lesson we're seeing across healthcare systems is that organizational bans alone are unlikely to be effective. Healthcare has always relied on multiple forms of governance, such as professional ethics and guidelines, national legislation, and disciplinary oversight through professional regulators or medical disciplinary boards. These different layers all influence how clinicians use new technologies. Moreover, the fact that clinicians are using AI tools despite those policies may suggest that these technologies meet a genuine clinical need."
"In this case, we should reconsider banning AI tools altogether and instead develop responsible use guidelines or safe alternatives."
Have new AI-related risks recently emerged that were not anticipated when the EU AI Act was being drafted?
"Yes. Technology has always evolved faster than law, so it is not surprising that new risks continue to emerge. The EU AI Act has anticipated this and was designed to be adaptable. For example, the list of high-risk AI use cases can be updated through delegated acts where new applications create significant risks to health, safety, or fundamental rights."
"Where entirely new categories of unacceptable risk emerge, the legislation itself can also be amended. A good example is the recent amendment of the EU AI Act (via the Digital Omnibus) to prohibit AI 'nudifier' applications."
The European Commission has postponed parts of the rules for high-risk AI systems, while the proposed Digital Omnibus package would soften some regulatory requirements. Do you think this is the right direction?
"Earlier versions of the Digital Omnibus would have substantially limited the application of the AI Act to AI-enabled medical devices by removing most of the AI-specific safeguards while still classifying these systems as high-risk. I was concerned about that approach because it would have weakened important safeguards relating to data governance, human oversight, and post-market monitoring."
"It’s positive for patients that this did not make it into the final Digital Omnibus on AI. Instead, it retains medical AI within the AI Act's high-risk framework while allowing targeted derogations where existing medical device legislation provides an equivalent level of protection. That strikes a much better balance between reducing unnecessary duplication and maintaining important safeguards for patients. At the same time, I don't think we should postpone the application of the high-risk rules for too long. Providing organizations with enough time to prepare is sensible, but prolonged delays risk creating uncertainty and postponing important safeguards."
Can Europe afford to introduce further AI regulation if it risks falling behind in the global AI race and becoming increasingly dependent on models developed in the United States or China? Would that ultimately undermine Europe's AI sovereignty?
"From an international perspective, I don't think the key question is whether Europe produces the largest number of AI systems. The more important question is whether we develop AI that is safe, effective, and actually improves patient care. We’re already seeing a growing number of European clinical AI tools emerge, designed around European healthcare systems, national clinical guidelines, and local regulatory requirements. That's important because a tool developed for the US healthcare system cannot simply be transplanted into Europe, where healthcare is organized very differently."
Which patient rights are currently most at risk when hospitals and healthcare providers deploy AI systems?
"First, the right to equal access to healthcare. AI systems don't always perform equally well for every patient group, and if they're trained on unrepresentative data or used in the wrong clinical context, they can reinforce existing health inequalities."
"Second, the right to information and informed decision-making. Patients should know when AI plays a meaningful role in their care, understand what it can and cannot do, and be able to discuss its recommendations with their clinician. Otherwise, it becomes much harder to make informed decisions about their treatment."
"Finally, patients need access to an effective remedy when something goes wrong because of an AI system. It can be extremely difficult for patients to identify who is legally responsible or prove causation. As AI becomes more complex, ensuring that patients can obtain explanations and receive compensation where appropriate becomes increasingly important."
What is your takeaway from the recent WHO conference 'Shaping AI in Health'?
One of the strongest themes at the WHO conference was that the conversation has shifted from whether AI belongs in healthcare to how we implement it safely, equitably and in ways that genuinely improve patient outcomes.
