Anchor Detection for RAG: Parallel Detectors, Then One LLM Call at the End

2026-06-24 · Source: Towards Data Science · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Intermediate, extended

Summary

The article "Anchor Detection for RAG: Parallel Detectors, Then One LLM Call at the End" details the retrieval component of an enterprise RAG system, focusing on how anchors are produced. It outlines a three-stage pipeline that runs keyword detection and embeddings in parallel on structured `line_df` and `toc_df` tables. This pipeline aggregates hits to structural units and concludes with a single LLM call for ranking candidates with reasons. Key principles include always running keyword detection (which is free), optionally running embeddings in parallel (costing microseconds with pre-computed indices), and deferring all LLM reasoning to a final arbiter call. The approach is demonstrated using the *Attention Is All You Need* paper, highlighting how it identifies relevant sections and lines for complex queries.

Key takeaway

For AI Engineers designing robust RAG systems, prioritize a hybrid retrieval strategy that integrates structured document data. You should implement parallel keyword and embedding detectors on both `line_df` and `toc_df`, deferring complex reasoning to a single LLM arbiter at the pipeline's end. This approach enhances auditability and precision, especially for enterprise documents where specific values and structural context are critical, outperforming generic BM25 or pure embedding methods.

Key insights

Enterprise RAG retrieval combines parallel keyword and embedding detectors, aggregating results for a single, auditable LLM ranking.

Principles

• Keyword detection is always-on and free.
• Embeddings run in parallel and are optional.
• One LLM call at the end for ranking.

Method

Stage 1: Parallel keyword and embedding detection on `line_df` and `toc_df`. Stage 2: Aggregate hits to structural units. Stage 3: Single LLM call ranks candidates with reasons.

In practice

• Boost lines with co-occurring keywords from semantic groups.
• Use regex for high-value patterns like monetary amounts or dates.
• Employ lexicons for enumerated entities (e.g., country names).

Topics

• RAG Retrieval
• Anchor Detection
• Keyword Search
• Embedding Similarity
• LLM Arbiter
• Document Intelligence
• Hybrid Retrieval

Best for: AI Engineer, Machine Learning Engineer, AI Architect

Related on AIssential

• RARE: Redundancy-Aware Retrieval Evaluation Framework for High-Similarity Corpora — Existing RAG benchmarks misrepresent real-world performance due to high document redundancy and similarity in enterprise corpora.
• Baseline Enterprise RAG, From PDF to Highlighted Answer — A minimal RAG pipeline can provide verifiable, sourced answers by structuring outputs and linking directly to document evidence.
• Hybrid Search and Re-Ranking in Production RAG — Combining hybrid search, cross-encoder re-ranking, and metadata filtering significantly improves RAG system accuracy for enterprise knowledge.
• I Stopped Using Libraries and Built TF-IDF From Scratch to Truly Understand Retrieval — Building TF-IDF from scratch clarifies the vector math underlying modern retrieval and RAG systems.
• Benchmarking RAG Architectures Locally on a Real Financial PDF — Local RAG on complex financial PDFs requires specialized extraction and architectural choices to achieve meaningful accuracy.
• Vespa AI and Surpassing the Limits of Vector Search — Vector similarity alone is insufficient for robust search relevance; tensor-based retrieval offers superior flexibility and efficiency.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Towards Data Science.