MamaBench: Benchmarking LLM Robustness in Maternal and Child Health Diagnosis through Counterfactual Clinical Perturbation

2026-07-15 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, AI in Healthcare · Depth: Expert, quick

Summary

MamaBench is introduced as the first counterfactual benchmark designed to evaluate large language model (LLM) robustness in maternal and pediatric AI diagnosis. It comprises 434 expert-authored clinical narratives, organized into 217 pairs across 371 pathologies. Unlike traditional benchmarks that assess questions in isolation, MamaBench measures a system's ability to distinguish clinically similar presentations requiring different interventions, using the Bias Trap Rate (BTR). Findings across eight configurations of four frontier LLMs reveal that base accuracy overstates robust accuracy by 16-28 percentage points. The paper also proposes Evidence-Anchored RAG (EA-RAG), a three-stage retrieval method. EA-RAG achieved a 20.3% BTR and 65.0% robust accuracy on Claude Sonnet 4.6, reducing BTR by 5.5 percentage points without degrading base accuracy. The remaining 20% BTR highlights counterfactual robustness as an ongoing challenge in clinical AI.

Key takeaway

For AI Scientists and Machine Learning Engineers developing clinical diagnostic tools, you should prioritize robust accuracy over base accuracy by integrating counterfactual evaluation into your LLM testing protocols. MamaBench demonstrates that models achieving high base scores can still fail to distinguish clinically similar cases. Consider implementing retrieval-augmented generation (RAG) methods like EA-RAG, which uses evidence coverage, to improve diagnostic robustness and reduce bias trap rates in critical healthcare applications.

Key insights

Current medical LLM benchmarks lack counterfactual evaluation, overstating diagnostic robustness in clinically similar cases.

Principles

• Isolated questions don't measure clinical distinction.
• Counterfactual evaluation reveals true robustness.
• Evidence coverage improves RAG for clinical AI.

Method

Evidence-Anchored RAG (EA-RAG) is a three-stage retrieval method: clinical parameter extraction, coverage auditing, and contrastive sub-queries, replacing aggregate similarity with an evidence coverage objective.

In practice

• Implement counterfactual clinical evaluations.
• Adopt EA-RAG's evidence coverage objective.
• Test LLMs with paired, similar clinical cases.

Topics

• LLM Robustness
• Counterfactual Evaluation
• Maternal & Child Health
• Clinical AI
• Retrieval-Augmented Generation
• Diagnostic Benchmarking

Best for: AI Scientist, Research Scientist, Machine Learning Engineer

Related on AIssential

• WHBench: Evaluating Frontier LLMs with Expert-in-the-Loop Validation on Women's Health Topics — LLMs significantly underperform on women's health clinical scenarios, particularly in safety and health equity, despite strong general medical benchmark results.
• Active Evidence-Seeking and Diagnostic Reasoning in Large Language Models for Clinical Decision Support — LLMs exhibit a substantial performance gap in active, multi-turn clinical diagnosis compared to static, full-context evaluations.
• Beyond Semantic Relevance: Counterfactual Risk Minimization for Robust Retrieval-Augmented Generation — CoRM-RAG enhances RAG robustness by optimizing for decision safety against user biases, not just semantic relevance.
• PSEBench: A Controllable and Verifiable Benchmark for Evaluating LLMs in Patient Safety Event Triage — LLM evaluation for high-stakes patient safety triage requires policy-grounded, interactive benchmarks beyond static classification.
• v333: Proceedings of CHIL 2026 — Machine learning, particularly LLMs, is rapidly expanding its utility across diverse clinical and administrative healthcare applications.
• Beyond Scalar Scores: Exploring LLM-based Metrics for Clinical Significance Evaluation in Radiology Reports — LLM-based metrics for radiology reports show discrimination bias, over-penalizing harmless variations while detecting errors.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.