Document Overlap Is Not Evidence Continuity: Measuring Retrieval Jitter in Citation-Based RAG Evaluation

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Advanced, quick

Summary

Punitha Ponnuraj's paper, "Document Overlap Is Not Evidence Continuity: Measuring Retrieval Jitter in Citation-Based RAG Evaluation," presents a diagnostic study challenging the assumption that retrieved evidence in RAG systems remains reproducible. The research investigates "retrieval jitter" by fixing queries, embeddings, and decoding while varying retrieval depth, chunk size, and overlap. Measuring evidence identity at both document (doc_id) and exact cited span (doc_id, span_hash) levels across BEIR ArguAna and SciFact datasets, the study reveals a consistent "Stability Gap." While document overlap remains moderate, span overlap frequently collapses, often showing total span turnover despite non-empty retrieval. This span-level instability is interpreted as a diagnostic of exact evidence-trace reproducibility issues, not semantic equivalence, motivating the inclusion of stability diagnostics in citation-based RAG evaluation metrics.

Key takeaway

For Machine Learning Engineers evaluating RAG systems, relying solely on document-level citations for evidence reproducibility is insufficient. This research demonstrates that exact cited spans can exhibit total turnover despite moderate document overlap, indicating significant "retrieval jitter." You should integrate span-level stability diagnostics alongside traditional citation-based metrics to accurately assess evidence-trace reproducibility. This ensures your RAG evaluations reflect true system reliability and prevent misleading conclusions about evidence continuity.

Key insights

Retrieval jitter causes significant instability in exact evidence spans, even when documents remain consistent, challenging RAG evaluation reproducibility.

Principles

• Evidence reproducibility is not guaranteed in RAG.
• Document overlap does not imply span continuity.
• Evaluate RAG stability at the span level.

Method

Measure "retrieval jitter" by varying RAG parameters (depth, chunk size, overlap) while fixing queries, embeddings, and decoding. Assess evidence identity at document and span levels.

In practice

• Implement span-level evidence tracking.
• Report stability diagnostics in RAG evaluations.
• Test RAG systems for retrieval jitter.

Topics

• RAG Evaluation
• Retrieval Jitter
• Evidence Reproducibility
• Span-Level Analysis
• BEIR ArguAna
• SciFact

Best for: Research Scientist, AI Architect, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Explicit Evidence Grounding via Structured Inline Citation Generation — FullCite improves LLM attribution by jointly generating document and evidence-span citations, revealing LLMs struggle with precise span localization.
• Does RAG Know When Retrieval Is Wrong? Diagnosing Context Compliance under Knowledge Conflict — Context-Driven Decomposition (CDD) measures and improves RAG's ability to resolve knowledge conflicts between retrieved context and parametric knowledge.
• NeocorRAG: Less Irrelevant Information, More Explicit Evidence, and More Effective Recall via Evidence Chains — Optimizing retrieval quality through "Evidence Chains" significantly improves RAG reasoning accuracy and efficiency.
• X-MADAM-RAG: Diagnosing and Handling Chinese-English Evidence Conflict in Retrieval-Augmented Generation — Multilingual RAG systems face significant challenges with contradictory evidence, requiring specialized diagnostic tools and pipelines like X-MADAM-RAG.
• Text Chunking Explained: Strategies, Overlap, and Semantic Splitting for Vector Search — Effective text chunking is crucial for RAG and semantic search, directly impacting retrieval precision and context relevance.
• ProvenAI: Provenance-Native Traces of Evidence in Generated Answers — Meaningful transparency in retrieval-grounded QA requires independently measuring correctness, citation fidelity, and source influence.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.