Reading Between the Lines: Toward Translating Verbose Patient-authored Messages into Clinician-Formulated Questions

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Advanced, quick

Summary

A lightweight LLM-based rewrite pipeline is proposed to translate verbose patient-authored messages into concise, clinician-formulated questions suitable for querying electronic health record (EHR) systems. This pipeline constrains outputs to 10-15 words and incorporates rule-based validation with regeneration. Evaluated on 140 distinct patient questions from the ArchEHR-QA dataset, the system demonstrates adherence to output length constraints. However, results indicate a tendency for the generated questions to be overly generic or tangential, with occasional hallucinations introducing unsupported clinical details. Automatic metrics, particularly BERTScore, show substantial overlap with human quality labels and serve as the strongest proxy for human preferences in pairwise meta-evaluation. The authors have released their code and annotations to facilitate further research.

Key takeaway

For NLP Engineers developing clinical communication tools, this research highlights the challenge of translating patient narratives into precise EHR queries. You should prioritize robust validation and regeneration steps in your LLM pipelines to mitigate generic or tangential outputs and prevent clinical hallucinations. Consider integrating metrics like BERTScore for automated quality assessment during development to align with human preferences and improve system reliability.

Key insights

Patient portal messages can be translated into EHR-queryable questions using constrained LLMs, though quality issues like genericness and hallucination persist.

Principles

• LLM output constraints improve conciseness.
• Rule-based validation helps refine generations.
• BERTScore correlates well with human judgment.

Method

A lightweight LLM-based rewrite pipeline translates patient narratives into 10-15 word clinician questions, using rule-based validation and regeneration to enforce constraints.

In practice

• Implement length constraints for LLM outputs.
• Use rule-based checks for factual accuracy.
• Employ BERTScore for automated quality assessment.

Topics

• Patient Portal Messages
• Clinical Question Generation
• Large Language Models
• Electronic Health Records
• Natural Language Processing
• BERTScore

Best for: AI Scientist, NLP Engineer, Research Scientist

Related on AIssential

• Possible or Definite? A Benchmark for Evaluating Diagnostic Uncertainty Preservation in Clinical Text — LLMs systematically distort diagnostic uncertainty in clinical text, often converting nuanced statements into definite assertions.
• Identifying Signs and Symptoms of Urinary Tract Infection from Emergency Department Clinical Notes Using Large Language Models — Prodigy is an efficient, scriptable annotation tool enabling data scientists to accelerate NLP development.
• Cross-Lingual Empirical Evaluation of Large Language Models for Arabic Medical Tasks — English-centric LLMs exhibit significant performance gaps and reliability issues in Arabic medical tasks.
• KliniskVestBERT: BERT Model Specialised to Norwegian Clinical Texts — Domain-specific pre-training significantly enhances BERT models for Norwegian clinical NLP tasks.
• Measuring the sensitivity of LLM-based structured extraction to prompt, model, and schema choices in clinical discharge summaries — LLM-based clinical data extraction sensitivity varies significantly with model and schema, especially regarding "absence-versus-silence" distinctions.
• ChatHealthAI: Aligning Electronic Health Record Representations with Large Language Models for Grounded Clinical Reasoning — ChatHealthAI integrates EHR representations with LLMs for grounded, interpretable clinical reasoning and accurate patient prediction.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.