Microsoft has published new research demonstrating that its AI Diagnostic Orchestrator can achieve substantially higher diagnostic accuracy than generalist physicians in simulated medical cases. The study, published in Science on 22 October, evaluates how artificial intelligence gathers clinical information, requests diagnostic tests and reaches conclusions through a sequence of decisions. Although the results are promising, the technology has yet to demonstrate its effectiveness in routine clinical practice.
The research builds on Microsoft's earlier work on AI-supported diagnosis, which we reported on in 2025. In the latest evaluation, the AI system correctly diagnosed approximately 83 percent of complex cases derived from the New England Journal of Medicine. Participating physicians achieved 36 percent accuracy when allowed to consult textbooks and online resources, compared with 22 percent without those resources.
A separate evaluation involving emergency department cases produced similar findings. The AI achieved approximately 80 percent diagnostic accuracy, compared with 47 percent among physicians. These comparisons involved simulated diagnostic scenarios rather than direct patient care. They therefore cannot establish whether the system would outperform clinicians in everyday healthcare settings.
AI follows the diagnostic process
The research centres on the Sequential Diagnosis Benchmark, or SDBench, developed to evaluate how AI systems make diagnostic decisions. Unlike conventional medical examinations, where all relevant information is presented simultaneously, the benchmark begins with a limited description of a patient's condition. The AI must determine which questions to ask, which investigations to request and when sufficient evidence is available to establish a diagnosis.
A separate component provides additional clinical information in response to these requests. When information is unavailable in the original patient record, the framework can generate synthetic responses. The system's final diagnosis is subsequently compared with the recorded diagnosis. Importantly, the benchmark also measures the costs associated with requested investigations. This allows researchers to assess whether an AI system reaches an accurate conclusion without unnecessarily increasing diagnostic testing.
Multiple AI roles examine clinical evidence
Microsoft's AI Diagnostic Orchestrator uses several simulated clinical roles to evaluate possible diagnoses and determine subsequent steps. These roles propose explanations, challenge conclusions and assess whether additional examinations would provide useful information. This approach is intended to reproduce aspects of clinical reasoning, where diagnostic decisions evolve as new evidence becomes available. The researchers have also made part of their evaluation infrastructure available through PhysioNet. The associated dataset includes 488 cases derived from emergency department encounters followed by hospital admission.
The dataset contains separate development, validation and clinician-reviewed testing components, allowing other researchers to investigate the methodology under applicable data-access conditions. However, the benchmark has limitations. Some responses are synthetically generated, and the scenarios cannot fully reproduce the complexity of real consultations, including communication difficulties, incomplete medical histories and unexpected clinical developments.
Prospective hospital evaluation planned
Microsoft is preparing to move beyond retrospective testing through a collaboration with Beth Israel Deaconess Medical Center in Boston. The planned prospective evaluation will involve patients interacting with the AI system in a hospital environment. Its performance will be assessed against independent evaluations by physicians. This next phase should provide evidence about how the technology performs when clinical information is collected directly from patients rather than reconstructed from existing records. It may also help establish how AI-supported diagnosis could fit into clinical workflows, including documentation, decisions about further investigations and physician oversight.
The study represents progress in evaluating medical AI beyond conventional question-and-answer benchmarks. Nevertheless, diagnostic accuracy alone is insufficient to establish clinical value. Healthcare organisations must also consider patient safety, unnecessary testing, integration with existing systems and responsibility for clinical decisions. The findings demonstrate the potential of sequential AI reasoning, but prospective clinical research will be necessary to determine whether that potential translates into safer, more effective or more efficient patient care.
Add ICT&health on Google
Show more content from ICT&health in Google Search.
