A Systematic Taxonomy of Failure Modes in Retrieval-Augmented Generation Systems

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A new systematic taxonomy identifies 33 distinct failure modes across 7 pipeline stages of Retrieval-Augmented Generation (RAG) systems: ingestion, representation, retrieval, generation, evaluation, deployment, and agentic orchestration. Developed from a structured literature review of 48 sources, this taxonomy provides formal definitions, observable manifestations, and evidence grading for each mode. The analysis highlights a significant research asymmetry, with representation, evaluation, and agentic orchestration failures being under-investigated despite their prevalence in production. Notably, 12 failure modes, including all 8 agentic ones, lack dedicated peer-reviewed empirical evidence, marking an "evidence desert" in a rapidly growing RAG deployment area. This work uniquely covers the full RAG pipeline, including agentic orchestration, with explicit evidence levels, surpassing prior enumerations.

Key takeaway

For MLOps Engineers deploying RAG systems, your current evaluation strategies likely miss critical failure modes, particularly in agentic orchestration. You should expand your testing to cover the 33 identified failure modes, especially those in representation, evaluation, and agentic stages, to improve system robustness and reliability in production environments. This proactive approach will mitigate poorly characterized failures.

Key insights

A systematic taxonomy categorizes 33 RAG failure modes across 7 pipeline stages, revealing critical research gaps in under-investigated areas.

Principles

• RAG systems fail in diverse, poorly characterized ways.
• Research attention is asymmetric across RAG pipeline stages.
• Agentic orchestration failures lack empirical evidence.

Method

A systematic taxonomy of 33 RAG failure modes was constructed via structured literature review of 48 sources, defining each mode with observable manifestations and three-level evidence grading.

In practice

• Evaluate RAG systems beyond single-stage metrics.
• Focus on representation and agentic orchestration failures.
• Consider the 33 defined failure modes for debugging.

Topics

• Retrieval-Augmented Generation
• RAG Failure Modes
• System Taxonomy
• Agentic Orchestration
• RAG Evaluation
• Literature Review

Best for: AI Architect, AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• Agentic RAG Failure Modes: Retrieval Thrash, Tool Storms, and Context Bloat (and How to Spot Them Early) — Agentic RAG's control loop introduces fragility, leading to predictable failure modes without proper budgeting and stopping rules.
• A Taxonomy of Runtime Faults in Model Context Protocol Servers — A new taxonomy empirically categorizes 73 runtime fault types in Model Context Protocol (MCP) servers, enhancing reliability understanding for LLM tool integration.
• RAG in Production: The Retrieval Failures Nobody Writes About — Production RAG systems require robust engineering to overcome subtle retrieval and context-use failures.
• The 80% AI Reliability Horizon — The 80% reliability horizon, critical for AI agent deployment, lags significantly behind the 50% benchmark due to compounding probabilistic failures.
• Does it Actually Work? Measuring Trust in a Pharma RAG System — Reliable RAG systems require robust monitoring and multi-agent safety layers to ensure accuracy and self-awareness.
• RAG Evaluation 101: What to Measure (and What Not to) — RAG evaluation must measure the gap between plausible and correct answers, requiring five distinct questions.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.