European AI Act still offers insufficient safeguards for healthcare

The European AI Act offers important protection against the risks of artificial intelligence, but does not guarantee that AI applications will align with the values of patients and healthcare professionals. This is the conclusion reached by Dutch researchers in a new study on DECIDE, an AI system designed to support GPs in identifying increased cardiovascular risks. According to the researchers, responsible AI in healthcare requires more than mere compliance with legislation: the doctor-patient relationship, shared decision-making and the practical usability of technology also deserve explicit attention.

The study was carried out by a multidisciplinary team comprising, amongst others, Leiden University, the LUMC, Delft University of Technology, Rotterdam University of Applied Sciences and the Dutch Patients’ Federation. The central question was to what extent the European AI Act actually protects the values that stakeholders consider important in medical AI.

DECIDE as a practical example

The researchers used DECIDE as a case study. This AI system is being developed to support GPs in the early identification of patients at increased risk of cardiovascular disease. DECIDE analyses existing medical data, for example, from electronic patient records, to highlight potential cardiovascular risks. Among other things, the system focuses on people who have not yet been diagnosed with cardiovascular disease but for whom preventive measures may be beneficial.

The basic principle is that AI does not make medical decisions independently, but provides information that GPs and patients can use when discussing health risks and possible next steps. The researchers analysed previously collected qualitative data from workshops involving various stakeholders. These included GPs, practice staff and patient representatives.

The values that emerged during these meetings were compared with the seven principles for trustworthy AI set out by the European expert group on artificial intelligence. The researchers then examined the extent to which these values are protected by the AI Act.

Legislation does not cover all healthcare values

The analysis shows that the AI Act contains important safeguards for, amongst other things, human oversight, transparency and the prevention of discrimination. Nevertheless, according to the researchers, these provisions offer only partial protection for what patients and healthcare professionals consider important. This is partly because the AI Act is strongly focused on product safety and the prevention of harm. However, within the healthcare sector, there are also values that are more difficult to translate into technical or legal requirements.

The researchers identify three additional areas of concern. The first concerns a positive approach to health and well-being. An AI system should not merely prevent patients from suffering harm, but should also make a genuine contribution to their health and quality of life. The second area of focus is the relationship between healthcare provider and patient. Shared decision-making, trust and personalised support are key components of good care. A technically reliable algorithm cannot automatically guarantee these aspects.

The third point concerns the context and practical applicability of AI recommendations. A risk assessment is of little value if a GP does not have sufficient time to discuss it or if appropriate follow-up care is lacking. In this way, the researchers distinguish between a system that meets legal requirements and an application that can actually function responsibly within everyday healthcare practice.

Increased workload and overdiagnosis

When an algorithm identifies large numbers of patients as being at risk, this can lead to additional consultations, diagnostic tests and referrals. Furthermore, if many alerts subsequently prove to be unfounded, there is a risk of causing unnecessary anxiety and overdiagnosis. Autonomy also plays an important role. Not every patient wishes to be constantly informed about potential future health risks. An AI application must therefore allow for the option to opt out of receiving additional information or follow-up actions. Furthermore, the use of patient data requires careful communication regarding privacy, consent and data processing.

The researchers point out that the interests of patients and healthcare professionals do not always fully align. A system that is attractive from a preventative perspective may, at the same time, place an additional burden on GPs or have undesirable consequences for patients. This is precisely why it is important that these different perspectives are taken into account right from the development stage.

Contractual agreements as a supplement

A notable recommendation from the new study concerns the role of contractual agreements between healthcare institutions and AI developers. According to the researchers, such agreements can provide additional safeguards for values that are not sufficiently explicitly protected by the AI Act. These might include agreements on providing comprehensible explanations of AI results, support in interpreting them, and the responsibilities of the parties involved.

These agreements do not replace existing legislation, but can help to ensure that the technology is better aligned with professionals’ duty of care and patients’ expectations. The researchers do emphasise, however, that their analysis is based on qualitative data from the Netherlands. The results cannot therefore be readily generalised to other countries or healthcare systems. Furthermore, the legal research focused primarily on regulations concerning product safety.

Responsible AI starts with practice

Human oversight is necessary, but not sufficient, if the technology does not adequately meet patients’ needs and the day-to-day practice of healthcare professionals. The key lesson, therefore, is that responsible medical AI is not solely a technical or legal challenge. The design of care processes, professional autonomy, patient preferences and the availability of follow-up care also determine whether an application actually adds value.

The European AI Act provides an important foundation for this. However, as the DECIDE study shows, reliable AI in healthcare will ultimately only emerge when legal protection is linked to the values and circumstances of the people who have to work with it.