10 Common RAG Mistakes We Keep Seeing in Production

2026-06-09 · Source: Towards Data Science · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Intermediate, extended

Summary

This analysis identifies ten common pitfalls observed in production Retrieval Augmented Generation (RAG) systems, categorized across four core "bricks": parsing, question parsing, retrieval, and generation. Key issues include treating structured documents like PDFs as flat text, leading to loss of tables and layout (Pitfalls 1-3), and failing to parse natural language questions into structured queries, which causes misinterpretation of scope and intent (Pitfalls 4-5). The article highlights over-reliance on vector databases for retrieval, neglecting keyword search and multi-granularity retrieval (Pitfalls 6-8). Finally, it addresses the lack of auditability in LLM generation, where raw text outputs lack verification flags or schema, and "not found" claims are trusted without deterministic proof of absence (Pitfalls 9-10). The article emphasizes that these mistakes lead to significant cost increases, such as \$131,000 annually for a 1200-page contract versus \$329 with a scoped pipeline, and reduced precision.

Key takeaway

For MLOps Engineers building or optimizing production RAG systems, prioritize structural parsing and question parsing upfront. Your pipeline's precision and cost-efficiency depend on treating documents as structured objects and questions as typed queries, rather than relying solely on embedding raw text. Implement hybrid retrieval and programmatic verification of LLM outputs to ensure auditability and prevent costly, silent failures at enterprise scale.

Key insights

Production RAG failures stem from ignoring document and question structure, over-relying on embeddings, and lacking generation auditability.

Principles

• Treat documents as structured objects, not flat text.
• Parse questions into typed objects with constraints.
• Employ hybrid retrieval, not just vector search.

Method

A robust RAG pipeline requires a structural parser, a typed question parser, hybrid retrieval at multiple granularities, and a generation verifier for programmatic checks.

In practice

• Implement relational parsers for structured documents.
• Use Pydantic schemas for question and answer validation.
• Combine keyword and embedding search for retrieval.

Topics

• RAG Systems
• Document Parsing
• Semantic Parsing
• Hybrid Retrieval
• LLM Auditability
• Production MLOps
• Cost Efficiency

Code references

• jxnl/instructor

Best for: AI Engineer, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• Baseline Enterprise RAG, From PDF to Highlighted Answer — A minimal RAG pipeline can provide verifiable, sourced answers by structuring outputs and linking directly to document evidence.
• RAG Is Not Dead. You’re Just Building It Wrong. — Optimizing RAG for production requires systematic engineering across all layers, prioritizing retrieval over LLM changes.
• Your RAG Isn’t Broken. Your Table Headers Are. — Stripped table headers silently destroy RAG retrieval quality by removing essential contextual schema from data.
• From Regex to Vision Models: Which RAG Technique Fits Which Problem — RAG problems demand tailored solutions based on document complexity and question control, not a universal "classic playbook."
• Why Naive Chunking Breaks RAG, and What to Build Instead — Structure-aware parsing and multimodal retrieval techniques are crucial for effective RAG on complex documents.
• Building a Production RAG Pipeline: Webhooks, Deduplication, and 40k Documents — Production RAG pipelines are primarily data engineering challenges, requiring robust solutions for continuous updates and data integrity.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Towards Data Science.