I Built a RAG System. Then I Added Evals. Here’s What I Found.

2026-07-27 · Source: Artificial Intelligence on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Intermediate, medium

Summary

A RAG system, initially appearing plausible, was found to be "lying" when subjected to rigorous evaluation. The author built an eval harness featuring a 17-question golden set and an LLM-as-a-judge to score relevance and faithfulness, logging every run in Postgres. This exposed critical issues: a 0.5 average relevance initially, revealing corpus coverage gaps (e.g., missing agent tool-calling docs, leader election) and prompt engineering flaws (model not utilizing all context). Introducing a local, in-process cross-encoder reranker ("Xenova/ms-marco-MiniLM-L-6-v2" via "@huggingface/transformers") initially dropped relevance to 0.88, but further analysis showed it exposed another pre-existing corpus gap. After addressing these data and prompt issues, the system achieved 0.94 average relevance and 1.0 average faithfulness.

Key takeaway

For AI Engineers building RAG systems, relying solely on "plausible" outputs is insufficient and misleading. You should integrate an LLM-as-a-judge eval harness early in development to scientifically measure relevance and faithfulness. This approach will quickly identify corpus gaps or prompt engineering issues, preventing wasted effort on refactoring functional retrieval code. Prioritize data ingestion and prompt refinement based on objective metrics to ensure robust system performance.

Key insights

Rigorous LLM-as-a-judge evaluation is crucial for RAG system reliability, exposing hidden data and prompt issues.

Principles

• "Plausible" RAG outputs are not metrics.
• Faithfulness 1.0 with low relevance indicates missing context.
• Cross-encoders improve reranking precision.

Method

The eval loop involves: golden question → RAG answer → LLM judge scores it → every run persisted for before/after comparison.

In practice

• Use LLM-as-a-judge for relevance and faithfulness.
• Log all eval runs for comparison.
• Implement local cross-encoder reranking.

Topics

• RAG Evaluation
• LLM-as-a-Judge
• Cross-Encoder Reranking
• Corpus Management
• Prompt Engineering
• pgvector

Code references

• bro-ankit/smart-semantic-booking-and-memory-engine

Best for: AI Engineer, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• 40 RAG Interview Questions and Answers — RAG enhances LLM accuracy and reduces hallucinations by providing real-time, verifiable external knowledge.
• UNH @ Rag4Reports: A Broad Exploration of LLM-Judges for RAG — Citation faithfulness and internal knowledge verification are key to effective LLM-as-a-Judge RAG evaluation.
• Building HITL Feedback RAG: Embeddings, Retrieval, and Reranking — HITL Feedback RAG improves LLM accuracy by dynamically injecting human-curated corrections via a robust retrieval and reranking pipeline.
• How I Stopped Guessing and Started Measuring My RAG Pipeline (LLM Zoomcamp 2026, Module 4) — Systematic evaluation of RAG retrieval using ground truth data and metrics like Hit Rate and MRR is crucial for performance optimization.
• The Context Dilemma: Prompting vs. RAG vs. Fine-Tuning — Prompting, RAG, and fine-tuning are distinct LLM techniques for managing context and grounding, not interchangeable solutions.
• How Do We Know AI Isn’t Lying? The Art of Evaluating LLMs in RAG Systems — Evaluating RAG systems requires multi-dimensional assessment of retrieval, understanding, grounding, and generation quality.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence on Medium.