MedDDC-Eval: Diagnosis-Decoupled Evaluation of Multi-Turn Medical Consultation Agents

2026-07-21 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Medical Devices & Health Technology · Depth: Expert, quick

Summary

MedDDC-Eval is a new diagnosis-decoupled testbed designed to evaluate multi-turn medical consultation agents more accurately. Traditional evaluation conflates the quality of policy-elicited patient history with the agent's terminal diagnosis generation, where strong generation can mask poor history collection. MedDDC-Eval addresses this by treating the elicited history as the primary comparison object and maintaining a constant history-to-diagnosis mapping via a shared frozen reader. It employs a grounded interface and a diagnosis-trajectory-efficiency (D/T/E) harness to measure diagnostic usefulness, information acquisition, and efficiency. Experiments showed that changing only the diagnostic reader shifted diagnosis F1 by 2.2-19.0 points and reversed 18% to 36% of policy orderings. Furthermore, post-training Qwen3-32B with Group Relative Policy Optimization (GRPO) using diagnosis and trajectory feedback improved performance by 9.7 and 4.6 total-score points on 100-case Record and 70-case Dialogue splits, respectively.

Key takeaway

For AI Scientists and NLP Engineers developing multi-turn medical consultation agents, traditional evaluation methods can obscure true agent performance by coupling history elicitation with diagnosis. You should adopt MedDDC-Eval's diagnosis-decoupled approach to gain clearer insights into your agent's evidence acquisition policy. This allows for more precise attribution of performance improvements and guides more effective policy development, as demonstrated by the Qwen3-32B training results.

Key insights

MedDDC-Eval decouples medical agent history elicitation from diagnosis for more accurate, interpretable evaluation.

Principles

• Decouple history elicitation from diagnosis generation.
• Use a shared frozen reader for consistent diagnosis.
• Measure diagnostic usefulness, acquisition, and efficiency.

Method

MedDDC-Eval uses a frozen reader to map elicited history to diagnosis. It employs a D/T/E harness for metrics and directional semantic coverage for open-ended item scoring. Group Relative Policy Optimization (GRPO) can post-train agents using diagnosis and trajectory feedback.

In practice

• Evaluate medical agents by isolating history quality.
• Apply GRPO with diagnosis and trajectory feedback.
• Use D/T/E harness for agent performance metrics.

Topics

• Multi-turn Medical Consultation
• Diagnosis-Decoupled Evaluation
• Medical AI Agents
• Qwen3-32B
• Group Relative Policy Optimization
• Natural Language Processing

Best for: AI Scientist, NLP Engineer, Research Scientist

Related on AIssential

• A safety-oriented hypothetico-deductive framework for AI-assisted differential diagnosis — AI diagnostic systems benefit from hypothetico-deductive reasoning and safety-oriented frameworks over one-shot predictions.
• A Multi-Agent Open-Source LLM for Structured Cancer Registry Information Extraction from Pathology and Medical Reports — A modular multi-agent LLM framework significantly improves context-dependent cancer registry data extraction from medical reports.
• Bridging the interpretability gap for medical artificial intelligence models using class-association manifold learning — CAML enhances medical AI explainability by decoupling decision patterns and visualizing global class-associated knowledge.
• MedBench: Deliberative Evaluation of Medical Language Models — Evaluating medical language models as deliberating agents reveals interaction-driven behaviors and limitations beyond single-model accuracy.
• Active Evidence-Seeking and Diagnostic Reasoning in Large Language Models for Clinical Decision Support — LLMs exhibit a substantial performance gap in active, multi-turn clinical diagnosis compared to static, full-context evaluations.
• Beyond the Individual: Virtualizing Multi-Disciplinary Reasoning for Clinical Intake via Collaborative Agents — Aegle virtualizes MDT reasoning for clinical intake using a multi-agent framework, improving diagnostic accuracy and documentation quality.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.