Our website uses cookies to improve and personalize your experience and to display advertisements (if any). Our website may also include third-party cookies such as Google Adsense, Google Analytics, and YouTube. By using the website, you agree to the use of cookies. We have updated our Privacy Policy. Click the button to view our Privacy Policy.

Does AI diagnose better than a doctor? The trial where the machine won on its own and did not improve the doctors

Does AI diagnose better than a doctor? The trial where the machine won on its own and did not improve the doctors
Contents
  1. The 30-second summary: the seven figures that matter
  2. The three-group trial
  3. Where precisely it fails
  4. Anchoring bias, measured in machines and in people
  5. Why the figures run from 25% to 97.8%
  6. What to do with this, in practice
  7. Frequently asked questions
  8. What remains unknown
  9. References

In a randomised clinical trial published in JAMA Network Open, fifty physicians worked through clinical cases with and without access to a large language model. Having the model in front of them did not improve their diagnostic reasoning: 76% against 74%, a two-point difference that does not reach statistical significance. The same model, working alone, scored 16 points above the physicians. That double result — the machine wins alone but adds nothing to the professional — is the most interesting thing this field has produced, and it defines the real problem: it is not about model capability, it is about how the model fits into a person’s work. This article walks through the five measurements that support it.

The 30-second summary: the seven figures that matter

Each row names the study and the year. Bear in mind that every single one of them evaluates written cases rather than actual patients.

FindingResultStudy and year
Physicians using the model compared to those without it76% versus 74%; a 2-point gap (CI: −4 to 8; p = 0.60)Goh et al., JAMA Netw Open (2024)
The model operating independently16 points higher than clinicians equipped with standard resourcesGoh et al., JAMA Netw Open (2024)
Inability to generate a differential diagnosisAn incidence exceeding 0.80 across all 21 models assessedRao et al., JAMA Netw Open (2026)
Inability to provide the correct final diagnosisUnder 0.40 within those identical modelsRao et al., JAMA Netw Open (2026)
Anchoring bias: placing the recommended diagnosis at the top55.6% for the models, alongside 21.2% for residents and 10.0% for attendingsSheppert et al., IJMI (2026)
Diagnostic precision spanning 30 studies and 4,762 casesRanging from 25% to 97.8%, featuring a significant risk of biasShan et al., JMIR Med Inform (2025)
Performance variance among models over 148 neurosurgery casesSpanning from 74% down to 36% depending on the specific modelWarrier et al., J Neurosurg (2026)

What you will find here

  • The randomized trial that evaluated the physician, the device, and the combination of both.
  • Where precisely the model fails: not on the diagnosis, but on the differential.
  • The experiment that measured anchoring bias in machines and in people.

• Why the meta-analytic range runs from 25% to 97.8%.

The three-group trial

Ethan Goh, Robert Gallo and a large team published a single-blind randomised clinical trial in JAMA Network Open in 2024. Fifty physicians took part — 26 attendings and 24 residents, with a median of three years in practice — from family, internal and emergency medicine, with 60 minutes to review up to six clinical cases. Randomisation decided whether, in addition to conventional resources, they had access to the language model.

Scoring used a validated rubric covering differential diagnosis accuracy, appropriateness of supporting and opposing factors, and next diagnostic steps, graded by blinded expert consensus.

The primary result: a median of 76% per case in the model group and 74% in the group without it, an adjusted difference of two percentage points (95% CI: −4 to 8; p = 0.60). Nor was there a significant difference in time per case.

The exploratory analysis that flips the headline

The researchers incorporated an additional comparative analysis to evaluate outcomes when the model independently addressed the cases without human assistance. Operating entirely on its own, the system outperformed the physician cohort relying on standard resources by a margin of 16 percentage points (95% CI: 2 to 30; p = 0.03).

Their conclusion is the right reading of the whole set: technology and workforce development are needed to realise the potential of physician-AI collaboration. The bottleneck is not model capability but the interaction.

StudyWhat it measuredWhat it foundWhat it cannot support
Goh et al., JAMA Netw Open (2024)Randomised trial with 50 physicians and clinical casesThe model did not improve the doctor; alone, it beat both groupsWritten cases, not real patients
Rao et al., JAMA Netw Open (2026)21 models across 29 cases, 16,254 responsesGood at final diagnosis, poor at the differentialThe graders were medical students
Sheppert et al., IJMI (2026)Diagnostic anchoring in 8 models and 25 physiciansModels anchor far more than peopleA forced-choice task, not open clinical practice
Shan et al., JMIR Med Inform (2025)Meta-analysis of 30 studies, 19 models, 4,762 casesAccuracy between 25% and 97.8%High risk of bias: cases already had known diagnoses
Warrier et al., J Neurosurg (2026)148 neurosurgery cases across five modelsFrom ChatGPT-4’s 74% to Gemini’s 36%Textbook cases, with queries run in May 2024

Where precisely it fails

In 2026, Arya Rao and co-workers released the most comprehensive assessment to date in JAMA Network Open: 21 general-purpose models, incorporating the newest iterations, evaluated over 29 standardised clinical cases, with 16,254 outputs graded three times across the sequential phases of the traditional clinical pathway.

The finding that matters is not the overall score but its breakdown. Failure rates exceeded 0.80 for differential diagnosis in every model, while for final diagnosis they stayed below 0.40.

Translated: these systems hit the correct diagnosis reasonably often, and they are bad at generating the list of alternatives that must be ruled out first. That list is precisely the clinical work, and what protects the patient when the obvious diagnosis turns out to be the wrong one.

The authors’ conclusion is explicit: despite version-based improvements, off-the-shelf models have not yet achieved the intelligence required for safe deployment.

Anchoring bias, measured in machines and in people

Anchoring is a classic mistake in clinical reasoning, defined by clinging to the initial hypothesis proposed. Alexander Sheppert and his research team quantified this phenomenon in 2026 within the International Journal of Medical Informatics through a straightforward methodology: nine pairs of internal medicine cases, each pair with one version containing a patient statement suggesting a plausible but unlikely diagnosis and one matched version without it.

Who is respondingRanks the suggested diagnosis firstIncludes it in the top five
Language models (8 evaluated)55.6% (40 of 72)97.2%
Internal medicine residents (20)21.2% (17 of 80)67.5%
Attending physicians (5)10.0% (2 of 20)65.0%
Mean anchoring-induced shiftModels 2.29; residents 0.88; attendings 0.87Significant difference (p = 0.008)
Relative odds against residentsOR 3.94 (95% CI: 2.32 to 7.33; p < 0.001)Against attendings, OR 4.07

The pattern is consistent and cuts against the usual intuition. AI is often presented as the antidote to clinicians’ cognitive biases; on this particular measurement, the models let themselves be anchored four to five times more often than an attending physician. And the difference between residents and attendings — less anchoring with more experience — is the expected one.

Why the figures run from 25% to 97.8%

Guxue Shan and his team released the most exhaustive systematic review and meta-analysis to date in JMIR Medical Informatics in 2025, encompassing 30 studies, 19 different models and 4,762 cases, while employing the PROBAST tool to evaluate potential bias risks.

The accuracy span concerning the initial diagnosis across every research paper’s top-performing model ranges from 25% to 97.8%, while triage classification results fluctuate between 66.5% and 98%. This discrepancy does not imply that certain algorithms outperform others; rather, the inherent complexity of the selected cases completely dictates the outcome.

The authors also flag the limitation that runs through this entire field: most studies carry a high risk of bias, and the main cause is the use of cases whose diagnosis is already known. Their conclusion is that the accuracy of these systems still falls short of clinical professionals.

The extent of divergence across models is effectively demonstrated by Akshay Warrier and his team in the Journal of Neurosurgery, drawing on 148 clinical cases from a neurosurgery manual: ChatGPT-4 achieved a 74% success rate, AtlasGPT managed 63%, ChatGPT-3.5 reached 53%, Copilot scored 48%, and Gemini recorded 36%. The majority of mistakes stemmed from omitting a crucial detail within the reasoning process, which was typically neuroimaging data.

What to do with this, in practice

Ordered by the strength of the evidence behind it:

• Do not assume that giving the tool to the clinician improves diagnosis. The single randomized trial available demonstrated no enhancement: a mere two-point variance, which lacked statistical significance.

  • Distrust the differential, not the final diagnosis. Failure rates exceed 0.80 on the differential and drop below 0.40 on the final diagnosis.
  • Do not introduce a hypothesis into the query. Models ranked the suggested diagnosis first in 55.6% of cases, against 10% for attendings.
  • Ask which cases any figure was measured on. The meta-analytic range, 25% to 97.8%, depends more on the difficulty chosen than on the model.
  • Avoid viewing models as interchangeable. Within a single batch of 148 instances, the performance disparity between the top and bottom performers reached 38 points.

Frequently asked questions

Can a language model diagnose better than a physician?

On written clinical cases, a model working alone can score higher. In the randomised trial published in JAMA Network Open in 2024, the model on its own scored 16 percentage points above the physician group with conventional resources (95% CI: 2 to 30; p = 0.03). That is not the same as diagnosing a real patient, where there is examination, testing and the course of the illness.

Does a doctor using AI diagnose better?

The only randomised trial available says not appreciably. The median diagnostic reasoning score was 76% with the model and 74% without it, an adjusted difference of two percentage points with a 95% confidence interval of −4 to 8 points (p = 0.60). Nor was time per case significantly reduced.

Where do these systems fail?

On the differential diagnosis, not the final one. In the evaluation of 21 models across 29 cases and 16,254 responses, failure rates exceeded 0.80 for generating the differential diagnosis in every model, while for the final diagnosis they stayed below 0.40. That is, they get the answer right and do not build the set of alternatives well.

Are they less prejudiced than humans?

On diagnostic anchoring, considerably more biased. In the 2026 forced-choice experiment, models ranked the patient-suggested diagnosis first in 55.6% of responses, against 21.2% for residents and 10.0% for attendings, with an odds ratio of around 4 against both groups.

Why do media outlets report such contrasting figures?

Because case difficulty changes the result entirely. The meta-analysis of 30 studies with 19 models and 4,762 cases puts primary diagnostic accuracy between 25% and 97.8%, and flags a high risk of bias in most of the work because cases with already known diagnoses are used.

What remains unknown

  • What happens with real patients. All of this work uses written cases; none measures diagnosis in clinic with incomplete information and an evolving picture.

• How to make the collaboration work. We know that handing the model to the doctor is not enough, but not what interface or training would change the result.

  • Whether anchoring can be corrected. The research records this within a closed exercise and explicitly urges replication across open clinical environments.
  • What happens with rare conditions. The case collections come from manuals, where uncommon diseases are overrepresented relative to real practice.

References

1. Goh, E., Gallo, R. et al. Large Language Model Influence on Diagnostic Reasoning. JAMA Network Open, 2024. DOI: 10.1001/jamanetworkopen.2024.40969

2. Rao, A. S., Esmail, K. et al. Large Language Model Performance and Clinical Reasoning Tasks. JAMA Network Open, 2026. DOI: 10.1001/jamanetworkopen.2026.4003

3. Sheppert, A., Shen, C. et al. Large language models exhibit greater diagnostic anchoring than physicians in a forced-choice vignette study. International Journal of Medical Informatics, 2026. DOI: 10.1016/j.ijmedinf.2026.106550

4. Shan, G., Chen, X. et al. Comparing Diagnostic Accuracy of Clinical Professionals and Large Language Models: Systematic Review and Meta-Analysis. JMIR Medical Informatics, 2025. DOI: 10.2196/64963

5. Warrier, A., Levin-Carrión, Y. et al. Comparative diagnostic capability of large language models in neurosurgery. Journal of Neurosurgery, 2026. DOI: 10.3171/2025.9.jns25846

This content is for information only and does not constitute medical advice. The figures cited come from studies carried out on written clinical cases, not on real patients, and none of them describes how these systems behave in clinical practice.

By Isabella Walker