Measuring the sensitivity of LLM-based structured extraction to prompt, model, and schema choices in clinical discharge summaries

2026-06-06 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Natural Language Processing · Depth: Advanced, short

Summary

The paper "Measuring the sensitivity of LLM-based structured extraction to prompt, model, and schema choices in clinical discharge summaries," submitted on June 4, 2026, investigates how Large Language Model (LLM) output for structured extraction from clinical free-text notes varies based on configuration choices. The study utilized MIMIC-IV v3.1 discharge summaries, employing a fixed schema of 17 clinical documentation flags (yes/no/not_documented) and a 47-tag vocabulary for primary admission reasons. Three prompt variants were tested across two model sizes. Cross-prompt agreement, measured by Cohen's kappa on ICD-stratified subsets, showed that for three-way flags, both models achieved similar pooled agreement (median kappa 0.69 and 0.68), with the larger model redistributing agreement. Collapsing the schema to binary resolved most cross-prompt disagreement, pinpointing the "absence-versus-silence" distinction as a key source. For multi-class admission categorization, changing the model reassigned the dominant tag on nearly half of notes, while prompt phrasing affected one in eight, and the larger model significantly reduced reliance on catch-all categories (from 44% to 26%). This research offers a reusable methodology for auditing extraction reproducibility at population scale.

Key takeaway

For NLP Engineers developing LLM-based clinical data extraction systems, you should carefully evaluate model and schema choices, as they significantly impact output sensitivity and reproducibility. Prioritize refining your schema to explicitly handle "absence-versus-silence" distinctions, which are a major source of disagreement. When dealing with multi-class categorizations, recognize that model selection has a greater impact than prompt phrasing on dominant tag assignments, and larger models can reduce reliance on generic categories.

Key insights

LLM-based clinical data extraction sensitivity varies significantly with model and schema, especially regarding "absence-versus-silence" distinctions.

Principles

• Model choice dominates multi-class categorization.
• Schema design impacts disagreement on absence.
• Larger models reduce catch-all category use.

Method

The study measured sensitivity by fixing the extraction task and varying one choice (prompt, model, schema) at a time, using Cohen's kappa for cross-prompt agreement and paired comparisons.

In practice

• Audit LLM extraction reproducibility at scale.
• Refine schemas to clarify "absence-versus-silence".
• Prioritize model choice for multi-class tasks.

Topics

• LLM Structured Extraction
• Clinical Discharge Summaries
• Prompt Engineering
• Model Sensitivity
• Schema Design
• Cohen's Kappa

Best for: AI Scientist, NLP Engineer, Research Scientist

Related on AIssential

• Measuring the sensitivity of LLM-based structured extraction to prompt, model, and schema choices in clinical discharge summaries — LLM extraction sensitivity varies significantly with schema design and model choice, especially on absence-versus-silence distinctions.
• Comparing LLM and Fine-Tuned Model Performance on NVDRS Circumstance Extraction with Varying Prompt Complexity — LLMs, guided by a context-aware "Complexity Score" for prompt selection, significantly outperform fine-tuned models on inferentially complex, low-prevalence data extraction.
• Cross-Lingual Empirical Evaluation of Large Language Models for Arabic Medical Tasks — English-centric LLMs exhibit significant performance gaps and reliability issues in Arabic medical tasks.
• An NLP-Driven Framework for Curriculum-Labor Market Alignment: Schema-Constrained LLM Extraction, ESCO-Anchored Semantic Matching, and Multi-Dimensional Gap Quantification — An NLP framework uses LLMs and semantic matching to quantify curriculum-labor market competency gaps with high reliability.
• Can LLMs Understand the Impact of Trauma? Costs and Benefits of LLMs Coding the Interviews of Firearm Violence Survivors — LLMs show promise for qualitative coding but struggle with relevance, data sensitivity, and ethical narrative erasure in trauma research.
• Supervision versus Demonstration-Based In-Context Learning for Multiword Expression Classification — Prompt sensitivity and demonstration quality critically impact LLM performance in metalinguistic classification tasks.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.