Building the Production LLM Pipeline RAG, Fine-Tuning, and Evaluation as Code Part-3

2026-07-19 · Source: Machine Learning on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Advanced, long

Summary

This article, Part 3 of a series on production LLM pipelines, details a full RAG architecture, addressing common failures where 80% of issues stem from ingestion and chunking, not the LLM itself. It advocates for structured chunking that respects document hierarchy over fixed-size token splits, and hybrid retrieval combining dense vector search with keyword methods like BM25 to overcome semantic search blind spots. The piece emphasizes reranking as the single biggest quality lever, using a cross-encoder pass on 20-50 retrieved chunks. It clarifies that RAG solves knowledge problems while fine-tuning addresses behavior, suggesting a hybrid approach for most production systems by 2026. Evaluation, particularly with RAGAS metrics (Faithfulness, Answer relevancy, Context precision, Context recall), is presented as an automated CI/CD gate, crucial for debugging and ensuring compliance with regulations like the EU AI Act.

Key takeaway

For MLOps Engineers building or optimizing RAG systems, prioritize robust offline pipeline components, especially structured chunking and metadata versioning, as 80% of failures occur here. Implement hybrid retrieval and a dedicated reranking step to dramatically improve answer quality and relevance. Integrate RAGAS evaluation metrics (Faithfulness, Answer relevancy, Context precision, Context recall) as automated CI/CD gates to proactively debug regressions and ensure compliance, rather than relying on manual checks. This approach shifts focus from LLM tuning to pipeline integrity for reliable production performance.

Key insights

80% of RAG failures originate in the ingestion and chunking layer, not the LLM, necessitating robust pipeline design.

Principles

• RAG failures are often invisible at the infrastructure layer.
• Chunking by document structure is critical for retrieval quality.
• Hybrid retrieval and reranking significantly improve RAG accuracy.

Method

A production RAG pipeline requires separate offline (ingestion, chunking, indexing) and online (query, retrieve, rerank, generate) components. Implement structured chunking, hybrid retrieval (vector + keyword), and a cross-encoder reranker. Integrate RAGAS evaluation as a CI/CD gate.

In practice

• Implement chunking based on document hierarchy (headings, paragraphs, sentences).
• Use hybrid retrieval (dense vector + BM25) for comprehensive search.
• Integrate a reranker for top 20-50 retrieved chunks to boost relevance.

Topics

• RAG Architecture
• LLM Production Pipelines
• Document Chunking
• Hybrid Retrieval
• Reranking
• RAGAS Evaluation

Best for: MLOps Engineer, AI Engineer, Machine Learning Engineer

Related on AIssential

• 10 Common RAG Mistakes We Keep Seeing in Production — Production RAG failures stem from ignoring document and question structure, over-relying on embeddings, and lacking generation auditability.
• RAG Is Not Dead. You’re Just Building It Wrong. — Optimizing RAG for production requires systematic engineering across all layers, prioritizing retrieval over LLM changes.
• Baseline Enterprise RAG, From PDF to Highlighted Answer — A minimal RAG pipeline can provide verifiable, sourced answers by structuring outputs and linking directly to document evidence.
• RAG in Production: The Retrieval Failures Nobody Writes About — Production RAG systems require robust engineering to overcome subtle retrieval and context-use failures.
• Building RAG Systems: A Complete Guide — RAG systems augment LLMs with external knowledge via structured ingestion and dynamic retrieval pipelines.
• How I Stopped Guessing and Started Measuring My RAG Pipeline (LLM Zoomcamp 2026, Module 4) — Systematic evaluation of RAG retrieval using ground truth data and metrics like Hit Rate and MRR is crucial for performance optimization.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning on Medium.