Does RAG Know When Retrieval Is Wrong? Diagnosing Context Compliance under Knowledge Conflict

2026-05-14 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, medium

Summary

A new study introduces Context-Driven Decomposition (CDD), an inference-time belief-decomposition probe designed to diagnose how Retrieval-Augmented Generation (RAG) models handle knowledge conflicts where retrieved context contradicts the model's parametric knowledge. CDD acts as an intervention mechanism for controlled retrieval conflict, revealing three key patterns across Epi-Scale stress tests, TruthfulQA misconception injection, and cross-model reruns. First, context compliance is measurable, with Standard RAG achieving only 15.0% accuracy on TruthfulQA misconception injection (N=500). Second, adversarial accuracy gains from CDD transfer across model families like Gemini-2.5-Flash and Claude Haiku/Sonnet/Opus, though the rationale-answer causal coupling does not, indicating different underlying mechanisms. Third, explicit conflict decomposition improves robustness against temporal drift and noisy distractors, with CDD reaching 71.3% on temporal shifts and 69.9% on distractor evidence on the full Epi-Scale benchmark. These findings establish context compliance as a structural axis for RAG probing, distinct from retrieval quality, and motivate the release of Epi-Scale for further systematic study.

Key takeaway

For AI Architects and Research Scientists evaluating RAG system robustness, this research highlights that standard RAG struggles significantly when retrieved context conflicts with internal knowledge, achieving only 15% accuracy in some adversarial settings. You should integrate tools like Context-Driven Decomposition (CDD) and benchmarks like Epi-Scale into your evaluation pipelines to systematically diagnose and improve context compliance, especially when deploying RAG in dynamic or adversarial environments where factual consistency is paramount. This approach helps ensure your RAG systems maintain accuracy under knowledge conflict.

Key insights

Context-Driven Decomposition (CDD) diagnoses how RAG models handle conflicting information between retrieved context and parametric knowledge.

Principles

• Context compliance is a measurable RAG vulnerability.
• Adversarial accuracy gains can transfer across model families.
• Explicit conflict decomposition enhances RAG robustness.

Method

Context-Driven Decomposition (CDD) is an inference-time belief-decomposition probe that intervenes on RAG models to diagnose context compliance under controlled retrieval conflict scenarios.

In practice

• Use CDD to measure RAG context compliance.
• Apply CDD for improved robustness against temporal drift.
• Evaluate RAG systems with Epi-Scale benchmark.

Topics

• Retrieval-Augmented Generation
• Context Compliance
• Knowledge Conflict
• Context-Driven Decomposition
• Epi-Scale Benchmark

Code references

• MLLMKCBENCH/MLLMKC

Best for: AI Architect, AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Does RAG Know When Retrieval Is Wrong? Diagnosing Context Compliance under Knowledge Conflict — Context-Driven Decomposition (CDD) measures and improves RAG's ability to resolve knowledge conflicts between retrieved context and parametric knowledge.
• Does RAG Know When Retrieval Is Wrong? Diagnosing Context Compliance under Knowledge Conflict — CDD diagnoses RAG's handling of knowledge conflicts, revealing context compliance and improving robustness across models.
• Your RAG System Retrieves the Right Data — But Still Produces Wrong Answers. Here’s Why (and How to Fix It). — RAG systems can confidently provide wrong answers when conflicting information is silently resolved during context assembly.
• FIDES: Faithful Inference via Deep Evidence Signals for Retrieval-Memory Conflict in RAG — Retrieval-memory conflict in RAG is token-level, requiring dynamic, deep-signal-driven contrastive decoding for faithfulness.
• Benchmarking RAG Architectures Locally on a Real Financial PDF — Part 3: The Measurement Problem — Multimodal RAG evaluation requires answer correctness against independent references, as text-grounded metrics misrepresent performance.
• Diagnosing LLM Arbitration Behavior over Pre-evidence Epistemic States in RAG-based Fact-Checking — LLM verifiers in RAG-based fact-checking show unreliable arbitration between parametric priors and contextual evidence, necessitating better evaluation and methods.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.