Building Cost-Efficient Agentic RAG on Long-Text Documents in SQL Tables

2026-02-18 · Source: Towards Data Science · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Advanced, long

Summary

This article presents an Agentic RAG (Retrieval Augmented Generation) architecture designed to operate directly on traditional SQL databases containing large documents in long-text fields, without requiring schema changes. The system addresses the challenge of combining structured computation, deep semantic understanding, and contextual insights from existing enterprise data. It utilizes a ReAct agent to orchestrate queries, intelligently deciding whether to use a SQL tool for computations and aggregations, a Vector tool for semantic search, or a hybrid approach combining both. The architecture was demonstrated using a subset of the Social Animal 10K Articles with NLP dataset, with `gemini-2.5-flash` as the LLM and `FAISS` for vector embeddings. Key design principles include careful management of tool docstrings versus system prompts and understanding the implications of pre- and post-filtering in vector databases.

Key takeaway

For AI Engineers and Data Scientists tasked with integrating LLMs into existing enterprise data infrastructure, this Agentic RAG architecture offers a robust solution. You can leverage your current SQL databases for both structured queries and semantic search without costly schema migrations. Pay close attention to the agent's routing logic and the choice of vector database filtering (pre- vs. post-filtering) to ensure reliable and accurate retrieval, especially for complex or hybrid queries.

Key insights

An Agentic RAG architecture can integrate LLMs with existing SQL databases for hybrid structured and semantic queries.

Principles

• Orchestrate SQL and semantic search for diverse queries.
• Align tool docstrings and system prompts to guide agent behavior.
• Pre-filtering vector databases enhance search accuracy.

Method

An Agentic RAG system employs a ReAct agent to route queries to specialized SQL and Vector tools. It uses metadata mirroring for filtering and handles computational, semantic, and hybrid queries by selecting the appropriate tool or combination.

In practice

• Mirror SQL metadata in vector stores for filtering.
• Use a test case repository to refine system prompts.
• Consider pre-filtering vector databases for precise results.

Topics

• Agentic RAG
• SQL Integration
• Semantic Search
• LLM Orchestration
• Vector Databases

Best for: Machine Learning Engineer, AI Engineer, Data Scientist

Related on AIssential

• Multi-Agent Architecture with RAG and Dynamic Context Windows for Text-to-SQL Optimization — A multi-agent Text-to-SQL system uses RAG and dynamic context to reduce token consumption.
• How to Build Agentic RAG with Hybrid Search — Hybrid search, combining vector and keyword methods, significantly enhances RAG system accuracy, especially for specific queries.
• Vectorless Agentic RAG Is Quietly Replacing Traditional RAG Architectures — Retrieval Augmented Generation (RAG) is evolving beyond vector databases, driven by the limitations of semantic similarity.
• I Just Built an Agentic RAG System From Scratch — Here’s What I Learned — Agentic RAG systems empower LLMs to autonomously refine searches, significantly improving answer quality and efficiency.
• Zero-Waste Agentic RAG: Designing Caching Architectures to Minimize Latency and LLM Costs at Scale — Two-tier caching with agentic validation significantly reduces RAG costs and latency while maintaining data freshness.
• Agent-Orchestrated Adaptive RAG: A Comparative Study on Structured and Multi-Hop Retrieval — Agentic RAG enhancements are domain-dependent and cost-intensive, necessitating selective, adaptive orchestration.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Towards Data Science.