Can AI Chatbots Reason Like Doctors?

2026-05-13 · Source: IEEE Spectrum · Field: Health & Wellbeing — Clinical Care & Medical Practice, Medical Devices & Health Technology, Health & Medical Research · Depth: Advanced, short

Summary

A study published 30 April in Science found that OpenAI's o1-preview large language model (LLM) outperformed physicians on several clinical reasoning tasks using real emergency room records. This finding comes amid mixed evidence on medical chatbots, with some studies showing impressive diagnostic performance while others document fabricated citations and flawed advice. OpenAI has already introduced ChatGPT for Clinicians and ChatGPT for Healthcare. The Science study, using data from 76 actual emergency room visits, reported the LLM achieved an "exact or very close diagnosis" 82% of the time at the final checkpoint, surpassing two physicians who scored 79% and 70%. Researchers emphasize significant limitations, stating AI does not replace doctors and that detecting LLM hallucinations is challenging due to their consistent convincingness. The medical community currently lacks a standardized scoring system for evaluating LLMs in clinical reasoning, leading to varied research conclusions. Experts advocate for moving from "AI versus humans" to understanding human-AI interaction and promoting responsible innovation.

Key takeaway

For medical professionals considering AI for clinical decision support, you should approach LLM integration with cautious optimism. While models like OpenAI's o1-preview show promising diagnostic accuracy, your primary focus must shift from "AI versus humans" to developing robust human-AI interaction workflows. Prioritize rigorous real-world testing and establish clear protocols for identifying and mitigating potential hallucinations, as these models can be convincingly wrong. Your goal should be to use AI as a supplementary tool for second opinions, not a replacement for your expertise.

Key insights

Large language models demonstrate potential in clinical reasoning, but require careful integration and standardized evaluation in medical workflows.

Principles

• LLMs can exceed human diagnostic accuracy in specific scenarios.
• AI should augment, not replace, physician clinical reasoning.
• LLM hallucinations are convincing and challenging to detect.

Method

Researchers compared LLM and physician diagnostic performance across multiple emergency room care stages using 76 real patient records.

In practice

• Seek LLM second opinions at critical diagnostic checkpoints.
• Conduct prospective clinical trials for medical LLM applications.
• Design workflows to minimize LLM-induced diagnostic errors.

Topics

• Clinical Reasoning
• Large Language Models
• Medical Diagnosis
• AI Hallucinations
• Clinical Decision Support
• OpenAI

Best for: CTO, VP of Engineering/Data, Director of AI/ML, AI Scientist, Research Scientist, Domain Expert

Related on AIssential

• Active Evidence-Seeking and Diagnostic Reasoning in Large Language Models for Clinical Decision Support — LLMs exhibit a substantial performance gap in active, multi-turn clinical diagnosis compared to static, full-context evaluations.
• Foundation Models Do Not Understand Biology — Foundation models lack biological understanding, necessitating guardrails for safe clinical AI applications.
• ClinicBot: A Guideline-Grounded Clinical Chatbot with Prioritized Evidence RAG and Verifiable Citations — ClinicBot provides trustworthy clinical support by prioritizing guideline evidence and preventing LLM hallucinations.
• Real-world safety and harms from patient-facing LLMs — Real-world data on AI chatbot harms in health is limited but growing, highlighting mental health risks.
• A domain-adapted large language model to support clinicians in psychiatric clinical practice — Domain-adapted LLMs can significantly enhance psychiatric clinical workflows, improving accuracy and efficiency.
• Please don’t trust your chatbot for medical advice — LLMs are unreliable for medical advice, frequently generating misinformation with overconfidence and lacking clinical reasoning.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by IEEE Spectrum.