Most RAG Hallucinations Are Retrieval Failures: How the Retrieval Brick Decides What the Model Can Invent

2026-07-15 · Source: Towards Data Science · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Intermediate, long

Summary

A recent analysis published in *Enterprise Document Intelligence* asserts that most RAG system hallucinations stem from retrieval failures rather than large language model generation errors. The study demonstrated this by querying the 55-page NIST Cybersecurity Framework v1.1 with "what backup practices keep data available after a ransomware attack?". A standard pipeline using SentenceTransformer("all-MiniLM-L6-v2") and cosine similarity ranked the correct answer on page 41 as 55th, the least similar page. This occurs because cosine similarity prioritizes surface meaning, averaging away critical keywords like "backup." The article identifies three retrieval failure modes: the answer not being retrieved, the wrong passage being retrieved (often surviving review due to plausible citations), and the correct answer being buried in distractors. The proposed solution involves anchoring retrieval on the correct span using signals like keyword matching and document structure, then tightly scoping the context. A simple keyword search for "backup" on the NIST document successfully moved the correct answer from rank 55 to rank 1.

Key takeaway

For MLOps Engineers troubleshooting RAG system hallucinations, you should first investigate your retrieval component before adjusting generation parameters. Measure the actual rank of the correct answer within your retrieved context; if it's low or absent, the issue is upstream. Implement multi-signal retrieval, combining keyword matching and document structure with embeddings, to accurately anchor and tightly scope the context provided to your LLM, preventing most "hallucinations" at their source.

Key insights

Most RAG hallucinations are retrieval failures, not LLM generation issues, caused by providing incorrect or insufficient context.

Principles

• Retrieval dictates what an LLM can invent.
• Cosine similarity often fails on enterprise documents.
• More context does not equal more grounding.

Method

Improve RAG retrieval by anchoring on the correct span using parallel detectors (keyword, structure, embedding) and then tightly scoping the context for generation.

In practice

• Measure answer rank before generation.
• Implement keyword matching for critical terms.
• Use document structure for context scoping.

Topics

• RAG Hallucination
• Retrieval-Augmented Generation
• Cosine Similarity
• Keyword Matching
• Document Intelligence
• NIST Cybersecurity Framework

Code references

• doc-intel/notebooks-vol1

Best for: Machine Learning Engineer, AI Engineer, MLOps Engineer

Related on AIssential

• Your RAG Isn't Hallucinating. Your Retrieval Is Lying. — RAG system "hallucinations" often stem from faulty retrieval providing incorrect context, not the LLM's generation.
• Hallucinations in LLMs: A Deep Technical Dive into Causes, Detection, and Mitigation — LLM hallucinations are inherent to their probabilistic nature, driven by training objectives that prioritize fluency over factual accuracy.
• Prompt Engineering Isn’t Enough: How Four Bricks of Context Engineering Stop RAG Hallucinations — RAG hallucinations primarily result from upstream context engineering failures, not the LLM itself.
• Why LLMs Fail and How RAG Makes AI Responses Smarter and More Reliable — RAG grounds LLM responses in external data, preventing hallucinations and ensuring up-to-date, accurate answers.
• Most RAG Hallucinations Are Extraction Errors: Seven Patterns for a Typed Generation Contract — RAG "hallucinations" are often extraction errors; fix them with a typed generation contract and structured validation.
• Your LLM Is Only as Good as What It Retrieves — Retrieval quality, not model scale, is the primary determinant of LLM factual accuracy and hallucination prevention in RAG systems.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Towards Data Science.