Evaluating Retrieval-Augmented Generation vs. Long-Context Input for Clinical Reasoning over EHRs

2018-03-12 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, extended

Summary

A study evaluated Retrieval-Augmented Generation (RAG) against long-context inputs for clinical reasoning using Electronic Health Records (EHRs) from hospitalized patients. Researchers from the University of Wisconsin-Madison, Loyola University Chicago, Boston Children's Hospital, Harvard Medical School, and the University of Colorado-Anschutz tested three prominent LLMs—o4-mini, GPT-4o-mini, and DeepSeek-R1—on three clinical tasks: extracting imaging procedures, generating antibiotic timelines, and identifying key diagnoses. The models processed up to 128K tokens. Findings indicate that RAG consistently matched or surpassed the performance of using only recent notes and nearly achieved the accuracy of full-context inputs while requiring significantly fewer input tokens. For imaging procedures, RAG showed at least a 3.75-fold performance gain. Antibiotic timelines also saw significant RAG improvements, though gains plateaued. Diagnosis generation, however, showed less consistent RAG benefits, with overall F1 scores not exceeding 44.41, suggesting task limitations.

Key takeaway

For Machine Learning Engineers implementing LLMs for clinical reasoning over Electronic Health Records, you should prioritize Retrieval-Augmented Generation (RAG). This approach offers near-parity performance with full-context models while drastically reducing token usage, making it more cost-effective and efficient. While RAG excels in extractive tasks like imaging procedure identification, carefully assess its benefits for complex, subjective tasks such as diagnosis generation, where performance gains may be less consistent. Consider fine-tuning retrieval parameters for optimal results.

Key insights

RAG provides efficient, competitive performance for clinical EHR tasks, often matching long-context LLMs with fewer tokens.

Principles

• RAG efficiency is highly task-dependent in clinical contexts.
• Targeted retrieval can approach full context performance with fewer tokens.
• EHR tasks vary in reasoning complexity and RAG benefit.

Method

Clinical notes were segmented into overlapping 128-token chunks, embedded using BGE-en-large-v1.5. Top-N (20, 40, 60) relevant chunks were retrieved via cosine similarity and inserted into the LLM prompt.

In practice

• Apply RAG for straightforward extractive EHR tasks like imaging procedure identification.
• Utilize RAG for temporal reasoning tasks such as antibiotic timeline generation.
• Carefully evaluate RAG's benefits for subjective tasks like diagnosis generation.

Topics

• Retrieval-Augmented Generation
• Clinical LLMs
• Electronic Health Records
• Token Efficiency
• Medical NLP Tasks
• Long-Context Models

Best for: NLP Engineer, AI Scientist, Research Scientist, Machine Learning Engineer

Related on AIssential

• When Cases Get Rare: A Retrieval Benchmark for Off-Guideline Clinical Question Answering — LLMs require retrieval augmentation and strong reasoning for reliable off-guideline medical question answering.
• Healthier LLMs: Retrieval-Augmented Generation for Public Health Question Answering — Retrieval-Augmented Generation (RAG) significantly improves LLM reliability and accuracy for public health question answering by grounding responses in official guidance.
• RAG Explained Through an Exam Analogy — Retrieval-Augmented Generation (RAG) grounds LLM responses in external, current data, reducing hallucinations and overcoming knowledge cutoffs without costly retraining.
• When Retrieval Doesn’t Help: A Large-Scale Study of Biomedical RAG — In biomedical QA, RAG offers minimal gains; the LLM's capacity to effectively use retrieved evidence is the primary bottleneck.
• A RAG Chatbot with Incremental Context Retrieval based on Local LLMs for Hospital Documents — Adaptive context control is crucial for reliable and efficient RAG systems using local LLMs in healthcare.
• Retrieval-Augmented Generation as a Bridge Between Classical NLP Pipelines and Modern LLM Systems — Retrieval-augmented generation (RAG) addresses the limitation of static knowledge in large language models (LLMs) by connecting them to external, up-to-date knowledge sources.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.