Researchers at Johns Hopkins University, in collaboration with the US FDA, have developed a new analysis tool capable of detecting hidden biases in the datasets underpinning medical AI systems. The technology is designed to prevent artificial intelligence from learning incorrect correlations from training data, which could lead to unreliable predictions that negatively impact patient care.
The researchers hope that the method will help developers, healthcare organisations and regulators to make medical AI more reliable and safer for clinical use. The results of the study, carried out in collaboration with the US Food and Drug Administration (FDA), have been published in npj Digital Medicine.
The ‘Clever Hans’ effect
AI models used in precision medicine learn to recognise patterns in large amounts of medical data. Although these models often make highly accurate predictions, they can also unintentionally establish correlations that have nothing to do with the underlying medical condition. The researchers refer to this as the so-called ‘Clever Hans’ effect. At the beginning of the last century, a horse named Hans appeared to be able to solve complex arithmetic problems by tapping his hoof. It later transpired that the animal had no grasp of mathematics, but was using subtle body language from his handler to determine the correct answer.
According to the researchers, AI models can pick up on misleading cues in a similar way. For example, an AI model designed to predict a person’s gender based on photographs of their eyes was found to rely primarily on the presence of mascara rather than biological characteristics. “We train models to be as predictive as possible, but we have little control over which features a model ultimately uses to arrive at a prediction,” says senior researcher Mathias Unberath of Johns Hopkins University.
Not just checking the model
To tackle this problem, the research team developed the Generalised Attribute Utility and Detectability-Induced Bias Testing (G-AUDIT) tool. Whereas existing methods mainly assess how an AI model performs after it has been developed, G-AUDIT focuses on the training data itself. The tool analyses datasets for hidden features that could lead an AI model to draw incorrect conclusions. G-AUDIT then ranks these attributes based on the risk that they may be a source of bias.
According to co-researcher Mitchell Pavlak, this marks a significant shift in the development of medical AI. Instead of checking retrospectively whether a model contains undesirable biases, the potential problems are identified whilst the data is being prepared.
The researchers tested the method on a variety of medical datasets, including imaging data, text files and spreadsheets. This revealed hidden differences in, amongst other things, the way data had been collected, the imaging equipment used and other contextual factors that say nothing about a patient’s health but may influence the AI’s predictions.
Clinical context misleads AI
One example involved a dataset of skin cancer images from two different clinics. One clinic mainly treated patients at high risk of skin cancer, whilst the other was a general dermatology practice with significantly fewer cancer cases. As both locations used different cameras, there was a risk that an AI model might come to regard the quality of the camera as a predictor of skin cancer. It also emerged that one of the clinics regularly placed rulers in photographs to track the growth of skin lesions over time. An AI model could therefore learn that the presence of a ruler is associated with skin cancer, whilst this is merely a consequence of that clinic’s working practices.
According to Unberath, such erroneous associations can have major consequences when AI systems are applied outside the original research environment, for example using photographs taken with a smartphone. As no ruler or specific camera is present in such cases, the predictions may become less reliable and could even lead to inequalities in healthcare.
The researchers aim to further expand G-AUDIT and expect that the method will eventually be applicable outside the healthcare sector as well. According to them, this shifts the focus from merely checking AI models to systematically assessing the quality and composition of the data on which those models are based. This should help to detect hidden biases at an earlier stage and further increase the reliability of AI applications in clinical practice.
Gen AI bias
Generative AI is becoming increasingly integrated into medical education, but experts warn that excessive reliance on these tools could weaken the clinical reasoning skills of future physicians. In december of last year we reported on an editorial in which researchers highlighted risks including automation bias, cognitive off-loading, deskilling, biased outputs, AI hallucinations, and privacy concerns.
They argue that medical education should place greater emphasis on assessing students' reasoning processes rather than just their final answers, while maintaining AI-free practical assessments that evaluate communication, physical examination, teamwork, and professional judgment. The authors also recommend making AI literacy a core component of medical training, enabling students to understand AI’s capabilities, limitations, and potential biases. Finally, they call on regulators, professional associations, and accreditation bodies to establish clear guidance on the responsible use of AI, ensuring that these technologies enhance rather than undermine medical education and patient care.
