MAX-EVAL-11: A Large Scale Benchmark for Evaluating Large Language Models on Full-Spectrum ICD-11 Medical Coding

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Health & Medical Research · Depth: Expert, quick

Summary

MAX-EVAL-11 is introduced as the first large-scale benchmark for evaluating Large Language Models on full-spectrum ICD-11 medical coding, addressing the absence of comprehensive tools for this global taxonomy transition. Comprising 10,000 MIMIC-III discharge summaries with expert-validated ICD-11 annotations, the benchmark covers 99.87% of the diagnostic taxonomy. It proposes a novel hierarchical evaluation framework that assigns partial credit based on ICD-11's 5-level structure, mitigating the brittleness of traditional exact-match metrics. Evaluations reveal significant performance gaps among state-of-the-art LLMs; Claude 4 Sonnet achieved the highest weighted score of 0.433, surpassing both general-purpose peers and specialized medical models like MedCoder. All models demonstrated near-zero exact match rates (0-4.8%), indicating the extreme difficulty of precise ICD-11 code generation and suggesting that broad reasoning capabilities currently outweigh domain-specific training for complex taxonomy scaling.

Key takeaway

For NLP Engineers developing automated medical coding solutions, recognize that precise ICD-11 code generation remains highly challenging for current LLMs. Your evaluation strategies should prioritize hierarchical metrics, like those in MAX-EVAL-11, over strict exact-match scores to reflect clinical utility and capture partial correctness. Consider general-purpose LLMs, as their broad reasoning capabilities currently offer better performance than narrowly specialized medical models for complex taxonomy scaling.

Key insights

Precise ICD-11 medical coding with LLMs is extremely difficult, requiring hierarchical evaluation for practical utility.

Principles

• ICD-11 coding demands hierarchical evaluation.
• General LLMs can outperform specialized medical models.
• Exact match metrics are brittle for complex taxonomies.

Method

MAX-EVAL-11 uses 10,000 MIMIC-III discharge summaries with expert-validated ICD-11 annotations. It employs a novel 5-level hierarchical evaluation framework for partial credit, moving beyond exact-match metrics.

In practice

• Evaluate LLMs using hierarchical metrics.
• Consider general LLMs for complex coding tasks.
• Prioritize reasoning over narrow domain training.

Topics

• ICD-11 Medical Coding
• Large Language Models
• LLM Evaluation
• Medical Benchmarks
• Hierarchical Evaluation
• MIMIC-III

Best for: AI Scientist, NLP Engineer, Research Scientist

Related on AIssential

• Class of LLMs: Benchmarking Large Language Models on the Brazilian National Medical Examination — Generalist LLMs often outperform specialized medical fine-tunes on high-stakes, language-specific medical exams.
• Blinded Multi-Rater Comparative Evaluation of a Large Language Model and Clinician-Authored Responses in CGM-Informed Diabetes Counseling — Retrieval-grounded LLMs can outperform clinicians in specific aspects of CGM-informed diabetes counseling.
• Beyond Problem Solving: UOJ-Bench for Evaluating Code Generation, Hacking, and Repair in Competitive Programming — LLMs can complement traditional online judges by identifying subtle code errors, but effective error detection remains computationally intensive.
• When Cases Get Rare: A Retrieval Benchmark for Off-Guideline Clinical Question Answering — LLMs require retrieval augmentation and strong reasoning for reliable off-guideline medical question answering.
• Analysing Lightweight Large Language Models for Biomedical Named Entity Recognition on Diverse Ouput Formats — Lightweight LLMs offer competitive biomedical NER performance, with specific output formats proving more effective than multi-format tuning.
• Linguistic Symptoms: Augmented Generation by Retrieval and Reasoning in LLMs under Portuguese-English Variation in Medical Contexts — Linguistic variation significantly impacts smaller LLMs in medical contexts, with larger models showing greater cross-language robustness.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.