An AI system developed by Microsoft has achieved substantially higher diagnostic accuracy than generalist physicians in a controlled medical challenge. The technology correctly diagnosed more than 80 percent of cases, compared with approximately 36 to 47 percent for participating doctors. Published in Science, the findings suggest that AI could become a powerful tool for clinical decision support. However, earlier research showing poor diagnostic performance by general-purpose chatbots highlights an important distinction: success in a structured simulation does not necessarily translate into reliable diagnosis in everyday healthcare.
Diagnosing complex medical conditions often requires clinicians to combine symptoms, medical histories, laboratory results and imaging findings. Microsoft's researchers developed the MAI Diagnostic Orchestrator (MAI-DxO) to investigate whether AI could perform this process systematically. Rather than generating an immediate answer, the system simulates a panel of medical experts that considers possible diagnoses, requests additional information and evaluates the evidence before reaching a conclusion.
Higher diagnostic accuracy
The researchers developed a testing framework called Sequential Diagnosis Benchmark (SDBench). It included 304 complex diagnostic cases published in The New England Journal of Medicine and 325 emergency department admission records. Each case was converted into an interactive diagnostic exercise. Physicians and AI initially received limited patient information and could subsequently request additional details or diagnostic tests.
An AI-based system supplied information from the original records, while another assessed the final diagnoses against the documented outcomes. MAI-DxO correctly diagnosed 83.2 percent of the complex published cases and 80.3 percent of emergency department cases. Generalist physicians achieved average accuracies of 36.1 and 46.8 percent, respectively. The 36.1 percent figure concerned doctors who could consult reference materials.
The AI also demonstrated a different diagnostic approach, requesting more information about patients' medical backgrounds before selecting tests. According to the researchers, this strategy could contribute to more efficient diagnostic investigations. However, potential cost savings depend on how the system is configured and require further evaluation.
Earlier research wich contrasting results
The findings appear to contradict research previously reported by us in May 2025. In that study, researchers at the University of Waterloo evaluated ChatGPT-4o using approximately 100 medical examination questions reformulated as open-ended diagnostic problems. The chatbot correctly identified the diagnosis in only 37 percent of cases. Moreover, almost two-thirds of its responses were considered unclear, regardless of whether the medical information was factually correct.
The researchers warned that plausible but inaccurate answers could create risks for patients attempting to diagnose themselves. Incorrect reassurance might delay necessary medical attention, while misleading warnings could cause unnecessary anxiety.
The contrast with Microsoft's results is striking, but the studies examined different technologies under different conditions. The Waterloo research evaluated a general-purpose chatbot responding to open-ended questions. Microsoft's system uses an orchestrated diagnostic process in which AI components repeatedly assess hypotheses and request additional evidence.
The case selections, assessment procedures and experimental settings also differed. Consequently, the percentages cannot be interpreted as a direct comparison demonstrating that diagnostic AI has improved from 37 to 83 percent within a year. Nevertheless, the findings suggest that how AI systems organise diagnostic reasoning may be as important as the underlying language model.
Specialist AI already shows diagnostic potential
Evidence of AI's potential in medical diagnosis is not limited to language models. In June 2024, we reported on the international PI-CAI study, coordinated by Radboudumc and published in The Lancet Oncology. Researchers evaluated AI algorithms developed to detect clinically significant prostate cancer in MRI scans. More than 200 AI teams and 62 radiologists from 20 countries participated. The researchers combined five leading algorithms into a single system and compared its performance with radiologists.
The AI detected almost seven percent more clinically significant prostate cancers and produced approximately 50 percent fewer false-positive findings than the radiologists in the comparison. These results demonstrated the potential of specialised AI systems to improve diagnostic accuracy and reduce unnecessary investigations. However, the technology still required further clinical validation before routine implementation.
Unlike Microsoft's broad diagnostic reasoning system, the PI-CAI algorithms focused on a clearly defined task involving medical images. Together, the studies illustrate that diagnostic AI encompasses substantially different technologies, each requiring evaluation appropriate to its intended clinical application.
Clinical validation
Despite Microsoft's encouraging results, the researchers explicitly caution against concluding that AI is already superior to doctors in clinical practice. SDBench evaluates diagnostic reasoning under controlled conditions, not direct patient care. It does not capture the full complexity of clinical examinations, communication, uncertainty or treatment decisions. The study also does not establish superiority over medical specialists or multidisciplinary teams.
Further research must determine whether combining AI with clinical expertise improves diagnostic accuracy, efficiency and patient outcomes in real healthcare environments. For now, the findings support a more nuanced conclusion than either unrestricted enthusiasm or outright scepticism. AI can perform impressively in specific diagnostic settings, while remaining unreliable in others. The challenge is to establish where these systems genuinely add value, how their limitations can be recognised and how clinicians can use them safely without surrendering professional judgement.
References
Science (research)
Add ICT&health on Google
Show more content from ICT&health in Google Search.
