Beyond Semantic Search: The Complete Guide to Advanced RAG with Milvus | Sagar Patil

2026-05-22 · Source: Data Science on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Software Development & Engineering · Depth: Intermediate, extended

Summary

This guide presents a comprehensive approach to building production-ready Retrieval-Augmented Generation (RAG) systems using Milvus as the vector database. It moves beyond basic semantic search, detailing an indexing pipeline that processes PDFs into dense and sparse embeddings using OpenAI's `text-embedding-3-large` model with a dimension of 1024, and BM25, respectively. The article demonstrates basic dense vector search and then introduces hybrid search, combining dense and sparse results with RRF (Reciprocal Rank Fusion) using a k=60 smoothing constant. It systematically layers on four advanced retrieval techniques: Metadata Filtering, Query Rewriting, Hypothetical Document Embeddings (HyDE), and Query Decomposition, each addressing specific RAG failure modes. Finally, it integrates Cross-Encoder Reranking using `ms-marco-MiniLM-L12-v2` to refine the top 5 results from an initial 20 candidates, emphasizing that retrieval quality directly impacts LLM answer quality.

Key takeaway

For AI Engineers building RAG systems for production, relying solely on basic semantic search is insufficient. You should implement a multi-stage retrieval pipeline, starting with Milvus's hybrid search (dense + sparse) as a baseline. Systematically integrate advanced techniques like metadata filtering for scope, query rewriting for conversational queries, or HyDE for vague questions. Always incorporate cross-encoder reranking on retrieved candidates to significantly improve the relevance and accuracy of context provided to your LLM, ensuring robust and reliable answers.

Key insights

Advanced RAG requires multi-stage retrieval, combining hybrid search with techniques like query rewriting and reranking for production reliability.

Principles

• Retrieval quality dictates LLM answer quality.
• Hybrid search (dense + sparse) improves recall.
• Cross-encoder reranking boosts final context precision.

Method

The guide outlines an indexing pipeline: PDF → Chunks (500 chars, 100 overlap) → Dense & Sparse Embeddings → Milvus Collection. Retrieval involves hybrid search, then optional advanced techniques and cross-encoder reranking.

In practice

• Fit BM25 on your specific corpus.
• Implement metadata filters for multi-document RAG.
• Rerank top-K candidates with a cross-encoder.

Topics

• Retrieval-Augmented Generation
• Milvus
• Hybrid Search
• Cross-Encoder Reranking
• Advanced Retrieval Techniques
• Vector Databases

Best for: AI Engineer, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• 10 Practical RAG Approaches: What Is Actually Useful and When to Use Each One — Effective RAG implementation requires a phased approach, starting simple and adding complexity only as specific bottlenecks are identified.
• Advanced Retrieval Pipeline for RAG (HyDE, Hybrid Search, Reranking) | Build 100% Local Retrieval — Combining HyDE, hybrid search, and reranking significantly enhances RAG retrieval accuracy and relevance.
• Implementing Hybrid Semantic-Lexical Search in RAG — Hybrid search, combining lexical and semantic methods via Reciprocal Rank Fusion, enhances RAG system retrieval accuracy.
• Building HITL Feedback RAG: Embeddings, Retrieval, and Reranking — HITL Feedback RAG improves LLM accuracy by dynamically injecting human-curated corrections via a robust retrieval and reranking pipeline.
• Issue #115 - Reranking in your RAG pipeline — Reranking enhances RAG pipeline precision by reordering initially retrieved documents using a Cross-Encoder.
• How Retrieval Systems Power Modern RAG Applications — RAG combines external retrieval with LLMs to overcome knowledge cutoffs and hallucinations, providing grounded, up-to-date responses.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Data Science on Medium.