The Untaught Lessons of RAG Retrieval: Cosine Is Not the Foundation

2026-07-03 · Source: Towards Data Science · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Advanced, medium

Summary

This article challenges the mainstream RAG retrieval approach (embed question, top-k by cosine, optional rerank). It proposes an alternative for enterprise RAG, treating retrieval as filtering on structured tables (`line_df` and `toc_df`) rather than free text search. The core idea is a three-signal parallel retrieval system: keyword search on lines, TOC reasoning, and embeddings as an optional fallback. This approach is detailed through six "untaught lessons": retrieval is filtering, not searching; anchor and context should be kept separate; embeddings come last; keywords prove absence while embeddings cannot; co-occurrence beats BM25 on narrow corpora; and one LLM pass over the TOC catches paraphrases. This method is supported by runnable companion notebooks on GitHub and is applicable across various sectors.

Key takeaway

For AI Engineers building enterprise RAG systems, abandon the default chunk-embed-cosine approach. Instead, treat retrieval as filtering on structured data like `line_df` and `toc_df`. Prioritize deterministic keyword searches and LLM-based TOC reasoning, reserving embeddings for vocabulary mismatches. This strategy improves auditability, precision, and cost-efficiency, ensuring more reliable answers and clearer absence proofs in narrow corporate datasets.

Key insights

Enterprise RAG retrieval should prioritize structured filtering and keyword signals, using embeddings as a last resort.

Principles

• Retrieval is filtering, not searching.
• Separate anchor from context for precision.
• Keywords prove absence; embeddings cannot.

Method

Implement a three-signal parallel retrieval pipeline: keyword search on `line_df`, LLM reasoning on `toc_df`, and embedding cosine as an optional fallback, with an LLM arbiter for final ranking.

In practice

• Use `line_df` and `toc_df` for structured filtering.
• Employ keyword search for deterministic absence proof.
• Apply LLM for TOC reasoning to catch paraphrases.

Topics

• RAG Retrieval
• Enterprise AI
• Document Intelligence
• Keyword Search
• Table of Contents Reasoning
• Embedding Models

Code references

• doc-intel/notebooks-vol1

Best for: AI Engineer, Machine Learning Engineer, AI Architect

Related on AIssential

• An LLM as arbiter in RAG retrieval: picking the right candidate with reasons — An LLM arbiter, given structured candidate briefs, can rank RAG results with auditable reasons, surpassing score fusion.
• Baseline Enterprise RAG, From PDF to Highlighted Answer — A minimal RAG pipeline can provide verifiable, sourced answers by structuring outputs and linking directly to document evidence.
• Your RAG System Isn’t Retrieving. It’s Guessing. — Current Retrieval Augmented Generation (RAG) systems largely rely on vector databases and the assumption that semantic similarity equates to relevance, a principle powering billions in infrastructure.
• The retrieval rebuild: Why hybrid retrieval intent tripled as enterprise RAG programs hit the scale wall — Enterprise RAG is shifting from simple vector search to complex hybrid retrieval for production agentic AI.
• RARE: Redundancy-Aware Retrieval Evaluation Framework for High-Similarity Corpora — Existing RAG benchmarks misrepresent real-world performance due to high document redundancy and similarity in enterprise corpora.
• RAG Is Not Dead. You’re Just Building It Wrong. — Optimizing RAG for production requires systematic engineering across all layers, prioritizing retrieval over LLM changes.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Towards Data Science.