Systematic Evaluation of the Quality of Synthetic Clinical Notes Rephrased by LLMs at Million-Note Scale

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Clinical Care & Medical Practice · Depth: Expert, quick

Summary

This study systematically evaluates the quality of synthetic clinical notes rephrased by Large Language Models (LLMs) from MIMIC databases at a million-note scale. The analysis, which includes intrinsic, extrinsic, and factuality evaluations, reveals that synthetic notes effectively preserve core clinical information and predictive utility for coarse-grained tasks. However, they demonstrate a loss of fine-grained details crucial for tasks like ICD coding. This detail loss can be substantially mitigated by rephrasing notes in chunks rather than as whole documents, though this approach may reduce factual precision under incomplete context. Error analysis identifies misinterpretation of clinical context, temporal confusion, measurement errors, and fabricated claims as dominant synthesis errors. Furthermore, these task-agnostic synthetic notes can augment task-specific training, particularly for rare ICD codes.

Key takeaway

For clinical NLP engineers developing LLM applications, understand that while LLM-rephrased clinical notes preserve coarse-grained information and predictive utility, they often lose fine-grained details critical for tasks like ICD coding. You should consider rephrasing notes by chunks to improve detail preservation, but rigorously fact-check for reduced factual precision due to incomplete context. These synthetic notes can effectively augment training data for rare medical codes.

Key insights

LLMs can rephrase clinical notes, preserving coarse details but losing fine-grained ones, which chunking can mitigate.

Principles

• Synthetic notes preserve core clinical information for coarse tasks.
• Fine-grained details are lost in LLM rephrasing.
• Chunk-based rephrasing can mitigate detail loss.

Method

Conduct systematic evaluation including intrinsic, extrinsic, and factuality. Rephrase clinical notes by chunks to preserve fine-grained details.

In practice

• Augment training for rare ICD codes with synthetic notes.
• Evaluate LLM-generated text for misinterpretation and factual errors.

Topics

• Large Language Models
• Clinical Notes
• Data Augmentation
• ICD Coding
• Factual Consistency
• Medical NLP
• MIMIC database

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Systematic Evaluation of the Quality of Synthetic Clinical Notes Rephrased by LLMs at Million-Note Scale — LLM-rephrased clinical notes retain coarse-grained utility but lose fine-grained detail, with chunking offering mitigation.
• The Slop Paradox: How Synthetic Standardization Erodes Clinical Uncertainty and Cross-Modal Alignment in AI-Rewritten Radiology Reports — AI-rewriting of clinical text for standardization can paradoxically degrade cross-modal alignment despite appearing cleaner.
• A Pipeline for Generating Longitudinal Synthetic Clinical Notes Using Large Language Models — A modular LLM-powered pipeline generates consistent, diverse longitudinal synthetic clinical notes to enable privacy-preserving healthcare AI development.
• Possible or Definite? A Benchmark for Evaluating Diagnostic Uncertainty Preservation in Clinical Text — Large language models poorly preserve diagnostic uncertainty in clinical text, posing a significant safety risk.
• Serialisation Strategy Matters: How FHIR Data Format Affects LLM Medication Reconciliation — FHIR data serialisation significantly impacts LLM medication reconciliation performance, varying by model size.
• Ten Headache Specialists versus Artificial Intelligence for Clinical Literature Summarization: A Critical Evaluation and Comparison — Expert-written clinical summaries remain preferred over RAG-LLM outputs, despite AI's growing proficiency and occasional indistinguishability.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.