Information-seeking failures of large language models in agentic clinical reasoning

2026-07-11 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Health & Medical Research, Clinical Care & Medical Practice · Depth: Expert, quick

Summary

A new agentic evaluation framework in hematologic oncology reveals that large language models (LLMs) struggle with information-seeking under uncertainty, despite achieving high scores on medical knowledge assessments. Across 32 frontier models, the best achieved only 68% overall accuracy. Information utilization, the fraction of available data actually requested, was the strongest predictor of diagnostic accuracy (R = 0.69, P < 0.001), yet utilization collapsed from 57% to 26% in the final round. Reasoning traces scored 91% above threshold on a clinical reasoning rubric but decorrelated from accuracy. Error analysis identified search satisficing, anchoring, and premature closure as dominant failure modes, mirroring cognitive biases in novice clinicians. This indicates LLMs' primary limitation in clinical oncology is systematic failure of information-seeking, not insufficient medical knowledge.

Key takeaway

For AI Scientists developing clinical decision support systems, you must prioritize active information-seeking capabilities over raw knowledge recall. Focus on designing models that proactively request and utilize critical data, especially molecular and cytogenetic information, to avoid diagnostic errors stemming from cognitive biases like premature closure and anchoring. Your systems should guide models to maintain high information utilization throughout the diagnostic process.

Key insights

LLMs fail in clinical reasoning due to poor information-seeking under uncertainty, not insufficient medical knowledge.

Principles

• Information utilization strongly predicts diagnostic accuracy.
• Locally coherent rationales do not ensure globally correct conclusions.
• LLMs exhibit cognitive biases similar to novice clinicians.

Method

An agentic evaluation framework requires models to proactively request clinical data across three sequential rounds before committing to a diagnosis and treatment plan.

In practice

• Evaluate LLMs on active information-seeking capabilities.
• Focus model training on uncertainty management strategies.
• Design systems to mitigate search satisficing and anchoring.

Topics

• Clinical Reasoning
• Large Language Models
• Agentic AI
• Diagnostic Accuracy
• Information Seeking
• Cognitive Biases

Best for: AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Active Evidence-Seeking and Diagnostic Reasoning in Large Language Models for Clinical Decision Support — LLMs exhibit a substantial performance gap in active, multi-turn clinical diagnosis compared to static, full-context evaluations.
• Please don’t trust your chatbot for medical advice — LLMs are unreliable for medical advice, frequently generating misinformation with overconfidence and lacking clinical reasoning.
• Compositional Reasoning Depth Predicts Clinical AI Failure: Empirical Evidence Consistent with Transformer Compositionality Limits in Electronic Health Record Question Answering — LLM accuracy in EHR QA systematically declines with increased compositional reasoning depth, quantified by "hop count."
• Agentic systems for breast cancer treatment recommendations — Agentic LLMs show promise for breast cancer recommendations but require expert oversight due to persistent clinical errors.
• RealICU: Do LLM Agents Understand Long-Context ICU Data? A Benchmark Beyond Behavior Imitation — RealICU is a new benchmark for LLMs in ICU settings, using hindsight-annotated data to overcome limitations of historical clinician actions.
• Evaluating Large Language Models on Misconceptions in Multi-Turn Medical Conversations — LLMs struggle to consistently correct medical misconceptions in multi-turn dialogues due to performance degradation and error propagation.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.