Microsoft’s MAI-DxO Scores Higher Than Doctors in Diagnostic Tests

Higher scores across two case groups

Microsoft’s MAI-DxO performed better than generalist physicians on two diagnostic benchmarks while estimating lower testing costs. The results, published in Science, make a strong case for testing AI as a diagnostic assistant. They don’t establish how well it would work in a clinic, where patient conversations, missing details and incomplete records shape the diagnosis. The gap between a benchmark and a working clinical tool is the central issue here.

Researchers evaluated the Microsoft AI Diagnostic Orchestrator, or MAI-DxO, on 304 clinicopathological conference cases from the New England Journal of Medicine and 325 emergency department-to-admission encounters from the MIMIC-IV-ED database. These are different kinds of cases: one set draws on published clinical discussions, while the other comes from emergency care records. Together, they give the model more than one setting in which to demonstrate its approach. But both are still documented cases presented for evaluation, rather than live encounters in which an AI has to collect information from a patient.

On the journal cases, the model reached 83.2% diagnostic accuracy. Generalist physicians given the same cases and access to reference materials scored 36.1%. For the emergency department encounters, MAI-DxO scored 80.3%, compared with 46.8% for the doctors. Those are large gaps, and they make the results hard to dismiss. They also describe performance on these particular benchmark cases; they don’t show that the model would outperform doctors across the full range of patients and conditions encountered in routine care.

The researchers also considered the estimated cost of tests used to reach a diagnosis. Microsoft said MAI-DxO’s diagnostic workups cost half to one third as much as those of the physicians, according to the company’s research blog post. Lower testing costs could matter if a system can reach sound conclusions without ordering unnecessary investigations. But the estimate belongs to this study’s evaluation, and it doesn’t by itself establish what a workup would cost in a clinic, where tests, staffing and patient needs vary.

A more realistic test, with limits

Earlier studies often gave AI systems a complete case description in one prompt. Here, MAI-DxO had to work through the case in stages: start with a patient’s initial complaint, form a provisional diagnosis, ask follow-up questions and choose tests while accounting for their costs. That process tests more than whether a model can recognize a diagnosis after receiving a finished summary. It also asks whether the model can decide what information it needs before settling on an answer.

That makes the benchmark a closer fit for diagnostic reasoning as it unfolds. The study’s authors said previous work hadn’t tested diagnosis as an active process that combines information gathering, accuracy and the cost of a workup. In real care, those elements are connected: a clinician’s next question can change which tests make sense, and a test result can send the diagnostic process in a different direction. A model that handles those choices well in a structured setting has cleared a more demanding bar than one given all the relevant facts at once.

But the patient exchange was simulated. MAI-DxO requested information from a “gatekeeper agent” that drew from the case record and supplied answers. In effect, one AI handled diagnosis while another mediated access to a prepared file. That setup can test how the model decides what to ask, but it can’t fully reproduce a patient describing symptoms, forgetting details or raising concerns a doctor might not anticipate. And this is the part most people overlook: the quality of a diagnosis depends partly on what the system can elicit, not just on how it reasons over the information it receives.

The benchmark also relied on detailed case records assembled through clinical work. That leaves a key practical question: whether an AI can gather useful information from real patients, with all the ambiguity and back-and-forth that entails. A patient may describe the same symptom in unexpected terms, be unsure when it began, or offer information only after a follow-up question. Those ordinary interactions can affect which possibilities a clinician considers. The study’s controlled setting leaves performance in actual care uncertain, a point also raised in criticism of AI doctor claims.

The numbers here are striking, but the comparison needs careful reading. The doctors and model received the same cases and reference access, which makes the benchmark useful for comparing their performance under those conditions. It still can’t answer whether the model would fit into a care team, how clinicians would respond to its recommendations, or how it would handle details that aren’t captured in a prepared record. Those questions require evidence from use beyond a case set.

What the results could mean for clinical use

This is more significant than it looks: the model’s reported advantage covers both diagnostic accuracy and estimated testing costs. That combination points toward a possible role in helping clinicians decide what to ask and which tests to consider, rather than simply producing a diagnosis from a completed chart. If you’re working in this space, the practical question is whether that support remains useful when the information arrives unevenly and the clinician has to weigh it alongside the patient’s account.

There’s also a difference between a system that performs well on a benchmark and one that earns a place in clinical practice. A benchmark can make cases and scoring consistent enough to compare results. A clinic adds communication, incomplete information and decisions that may change as the conversation unfolds. MAI-DxO’s results support further evaluation of that kind of diagnostic assistance, but they don’t resolve how it should be used or how its recommendations would affect care.

Microsoft forecasts clinical deployments

Microsoft AI chief Mustafa Suleyman told The Verge that “extremely impressive deployments of intelligent systems in production” would arrive the following year. He added that he expects AI to change how care is delivered, according to the interview. That’s a forecast from a company executive, while the study reports results from structured evaluations. The two claims belong in the same conversation, but they aren’t the same kind of evidence.

For now, MAI-DxO’s results show what a diagnostic model can achieve on carefully documented cases under structured conditions. The score and cost estimates are promising enough to justify closer scrutiny, though they don’t settle the question of clinical performance. The next test is whether those results carry over to conversations with patients, where the information may be unclear, incomplete or revealed only over time. That’s the more consequential measure.

More on medical AI: Startup Deploys AI-Engineered Virus on Human Patient

The post Microsoft Says Its New AI Is Better at Diagnosing Patients Than Real Doctors appeared first on Futurism.