🔍 Read the full analysis: What Microsoft’s AI Study Says About Comparing AI With Doctors on ThorstenMeyerAI.com
Get the little things that make your day delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
A paper published in Science on Oct. 22 reports that Microsoft’s MAI-DxO system reached 83.2% diagnostic accuracy on 304 complex case records, compared with 36.1% for generalist physicians using reference materials. The comparison used historical, text-only cases and did not include specialists or real patient care; Microsoft says a prospective study is planned at Beth Israel Deaconess Medical Center.
Microsoft’s MAI-DxO diagnostic system scored 83.2% on 304 difficult medical case records, compared with 36.1% for generalist physicians using textbooks and the internet, according to a Science paper published October 22. The study reports a large performance gap on selected cases, but it used historical text records rather than real patient encounters and did not compare the system with specialists or medical teams.
The paper, titled Sequential diagnosis with language models, evaluates Microsoft’s system on cases drawn from the New England Journal of Medicine. On a separate set of 325 emergency department cases, MAI-DxO scored 80.3%, while physicians scored 46.8%, according to Microsoft’s reported results. The study also reports physician accuracy of 22.3% on the complex cases when doctors could not use reference materials.
The test was designed to assess how a participant gathers information, not only whether it can select a diagnosis after seeing a complete file. Each case began with a short summary. The AI or physician had to request additional details, such as answers to questions or test results, from a simulated gatekeeper holding the case record. The benchmark scored both the final diagnosis and the estimated cost of obtaining information.
Microsoft describes MAI-DxO as an orchestration system that runs a language model as a panel of virtual physicians. The agents propose diagnoses, request further information and challenge each other over multiple rounds. In the company’s chart, GPT-5 without the MAI-DxO setup scored 78.9% on the complex cases, close to the system’s 83.2%; Microsoft estimated testing costs at $8,409 per case for GPT-5 alone. The results suggest much of the measured diagnostic performance came from the underlying model, while the orchestration layer added several percentage points and reduced estimated test costs.
Infographic 01 · The scoreboard
83% vs. 36%: the number, and who it was measured against.
Microsoft’s diagnostic system beat individual generalist physicians by a wide margin on a simulated, step-by-step test. The comparison group decides how much that says.
Hard cases (NEJM)
Diagnostic accuracy, 304 case records
Physician averages rest on 662 (open book) and 764 (closed book) case attempts, per Microsoft’s chart data.
Who was compared
The fine print behind “beats doctors”
- Generalists only: primary care doctors and hospital generalists. No specialists, no teams.
- Working alone: no colleagues; in the preprint also no search engines or language models.
- Text-only, historical cases: no real patients, no physical exam, no imaging.
- Hard by design: rare teaching cases; no healthy patients, so false alarms were not measured.
- Self-reported: all 15 authors are listed with Microsoft affiliations; peer-reviewed, not independently replicated.
Sources: Science 394, 423–428 (22 Oct 2026); Microsoft AI, “Building Medical Superintelligence” (22 Oct 2026) and its charts; physician cohort details from the arXiv 2506.22405 preprint (2025).
Powered by Thorsten Meyer AI
Why the Test Design Matters
The study matters because it tests a feature of clinical reasoning that standard multiple-choice evaluations often leave out: choosing what to ask or test next. If the approach can be validated in clinical settings, a system that gathers relevant information efficiently could have a role in supporting clinicians as they work through difficult cases.
But the accuracy figures do not show that MAI-DxO can replace doctors or improve outcomes for patients. Participants worked from text-only historical cases, without physical examinations, direct access to imaging or responsibility for treatment. The physician comparison group consisted of generalists working individually, while the AI system used a panel of agents. The study therefore measures performance under a particular benchmark design, not how the system compares with the specialist teams that may handle unusually difficult cases in practice.
The case selection also affects how to interpret the scores. The source account says the NEJM set contained challenging published cases and no healthy patients or harmless explanations. That limits what the results can say about false alarms in routine care. A system may perform well at identifying rare conditions in a set chosen for diagnostic difficulty without showing how often it would raise unnecessary concerns among a broader patient population.
AI diagnostic tools for healthcare
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
From Preprint to Science Paper
An earlier version of the research appeared on arXiv in June 2025 as a non-peer-reviewed preprint. That version reported an 80% score for the AI, then running on OpenAI’s o3 model, against 20% for doctors. The Science paper appeared on October 22, 2026, after submission in April and acceptance in August, according to the source material. It lists 15 authors affiliated with Microsoft, including Microsoft AI chief executive Mustafa Suleyman.
The published paper’s appearance in a peer-reviewed journal means the work passed the journal’s review process; it does not establish that independent researchers have reproduced the results. The source account notes that the full Science article is behind a paywall and that some methodological details cited in its discussion come from the earlier preprint, so they may not match the final paper exactly.
In that earlier account, physicians were generalists from the United States and United Kingdom, and the AI and doctors were assessed on different numbers of cases. The source says the doctors were scored on the 56 most recent held-out cases, while the AI was tested on all 304. It also describes a model-based grading process and estimated test prices drawn from billing codes and a 2023 U.S. hospital price list. Those details should be read as descriptions of the preprint unless confirmed in the full published paper.
“I really believe that in 2027, we will be on the verge of a coding style breakthrough for medical records.”
— Microsoft AI chief executive Mustafa Suleyman, speaking to TechCrunch
medical AI decision support systems
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Limits of the Clinical Evidence
The study does not establish how MAI-DxO would perform with real patients, complete clinical records or the practical pressures of a hospital. It also does not show whether using the system changes treatment decisions, patient outcomes, diagnostic delays or costs in ordinary care. Those questions require evidence beyond a retrospective case benchmark.
Some comparison details need care. The AI system used multiple virtual agents, while the physician group consisted of individual generalists; specialists and physician teams were not included, according to the source account. The cases were selected for difficulty, and the source says they did not include healthy patients or benign explanations, preventing an assessment of false alarms. The study’s reported accuracy figures therefore should not be treated as a measure of performance across all patients.
It is also not yet clear from the information provided whether independent groups have replicated the findings, how the final Science paper’s full methods compare with the preprint, or how reliable the estimated test costs would be in a clinical workflow. Peer review validates neither generalizability nor real-world benefit on its own.
AI-powered medical diagnosis software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Planned Study With Patients
Microsoft says the next step is a prospective study at Beth Israel Deaconess Medical Center in Boston. According to the source account, consenting patients are expected to interact with the system alongside their normal care. That setting could provide evidence about how the tool performs when people and clinicians are involved, rather than only archived case records.
Details supplied here do not specify the study’s start date, size, design, outcome measures or whether results will be independently evaluated. Until those details and findings are available, the Science paper supports a narrower conclusion: MAI-DxO scored higher than the studied generalists on the selected benchmarks, while its safety, usefulness and performance in routine clinical care remain unestablished.
As an affiliate, we earn on qualifying purchases.
Key Questions
What did Microsoft’s study report?
In the Science paper, MAI-DxO scored 83.2% on 304 complex case records. Generalist physicians scored 36.1% with textbooks and internet access, and 22.3% without those resources, according to the reported results.
Did the study test MAI-DxO on patients?
No. The benchmark used historical, text-only medical cases in a simulator. Microsoft says a separate prospective study involving consenting patients is planned at Beth Israel Deaconess Medical Center.
Were specialists included in the comparison?
According to the source account, the comparison group consisted of generalist physicians working individually. Specialists and physician teams were not included, limiting what the results say about comparisons with real-world specialist care.
Does peer review mean the results have been independently confirmed?
No. Publication in Science means the paper went through peer review, but the source material says independent replication has not been established. Peer review alone does not show that the results generalize to clinical practice.
What happens next?
Microsoft says it plans a prospective study at Beth Israel Deaconess Medical Center, where consenting patients would interact with the system alongside their usual care. The available information does not provide the study’s timetable or design details.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
