Active Evidence-Seeking and Diagnostic Reasoning in Large Language Models for Clinical Decision Support

2026-05-23 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Healthcare Systems & Policy, Clinical Care & Medical Practice · Depth: Expert, extended

Summary

A new study introduces ROUNDS-Bench, an OSCE-inspired standardized patient simulator and controlled benchmark designed to evaluate large language models (LLMs) in active diagnostic inquiry for clinical decision support. Across 468 cases and 15 models, the research found that multi-turn evidence seeking significantly reduces diagnostic accuracy by 12.75% and lowers supporting-evidence quality by 24.36% compared to full-context evaluation. Error analysis attributes these declines to premature diagnostic closure and inefficient questioning strategies by LLMs. These findings suggest that current static, full-context benchmarks may substantially overestimate LLM performance in dynamic, interactive clinical settings, highlighting the critical need for complementary interactive assessments to ensure safer and more effective medical AI agents.

Key takeaway

For AI Scientists and Machine Learning Engineers developing clinical decision support systems, you must move beyond static benchmarks. Your LLM evaluations should prioritize interactive evidence-seeking and robust evidence grounding, as current models show a 12.75% accuracy drop and 24.36% evidence quality decline in dynamic settings. Focus on developing models that can actively justify diagnoses with verifiable evidence to mitigate "hallucinated reasoning" and ensure auditability for safer clinical deployment.

Key insights

LLMs exhibit a substantial performance gap in active, multi-turn clinical diagnosis compared to static, full-context evaluations.

Principles

• Static benchmarks overestimate LLM clinical performance.
• Active diagnosis requires hypothesis-driven inquiry, not just pattern completion.
• Evidence quality often degrades faster than diagnostic accuracy.

Method

ROUNDS-Bench uses an OSCE-inspired Standardized Patient Simulator with an Information Gating Mechanism and a dual-task evaluation (full-context vs. active inquiry) across 468 cases and 15 LLMs, scored by an independent LLM arbitrator.

In practice

• Implement interactive assessments for medical AI.
• Penalize unsupported "correct" diagnoses in LLMs.
• Integrate RL from clinician feedback for evidence-based reasoning.

Topics

• Large Language Models
• Clinical Decision Support
• Standardized Patient Simulation
• Interactive Diagnostic Reasoning
• Medical AI Evaluation
• Evidence Quality

Code references

• Leonard-zc/ROUNDS-Bench

Best for: AI Product Manager, AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• v333: Proceedings of CHIL 2026 — Machine learning, particularly LLMs, is rapidly expanding its utility across diverse clinical and administrative healthcare applications.
• Can AI Chatbots Reason Like Doctors? — Large language models demonstrate potential in clinical reasoning, but require careful integration and standardized evaluation in medical workflows.
• Possible or Definite? A Benchmark for Evaluating Diagnostic Uncertainty Preservation in Clinical Text — Large language models poorly preserve diagnostic uncertainty in clinical text, posing a significant safety risk.
• AIPatient Arena: EHR-grounded evaluation of large language models in end-to-end clinical consultation workflows — AIPatient Arena reveals LLMs need process-based evaluation beyond final answers for clinical readiness, highlighting interaction failures.
• T2D-Bench: Evidence-Gated Evaluation of LLM Outputs for Type 2 Diabetes Using a Multi-Layer Clinical-Lifestyle Knowledge Graph — LLMs require evidence-gated evaluation against clinical knowledge graphs to ensure medical accuracy and guideline compliance.
• Statistics, Not Scale: Modular Medical Dialogue with Bayesian Belief Engine — Strictly separating language processing from probabilistic reasoning in medical AI yields superior, auditable, and private diagnostic systems.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.