How we taught agents to use good retrieval - Hanna Lichtenberg, Mixedbread AI

2026-07-07 · Source: AI Engineer · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Advanced, long

Summary

Mixedbread AI addresses the "knowledge gap" between the rapidly advancing reasoning capabilities of large language models (LLMs) and the slower evolution of retrieval systems. Benchmarks like BroSque Plus and Office QA Pro demonstrate a significant performance drop (e.g., 93% Oracle vs. 9% with Codex on BrowseComp) when LLMs use default search tools in noisy corpora, indicating retrieval, not reasoning, is the bottleneck. LLMs generate poor search queries due to training biases towards coding tasks (grep), mimicking human web queries, and benchmark styles favoring BM25. Mixedbread developed a search agent with a powerful harness featuring four tools: overview, main semantic, filter chunks, and grep. This agent employs an iterative loop with parallel searches and is trained using supervised fine-tuning and on-policy reinforcement learning with a combined retrieval and trajectory reward. Intermediate results show the agent achieving an NDCG@10 of 0.4 on Oblique Congress (vs. 0.18 for GPT multi-hop) and 93.4% accuracy on Snowflake's Match QA benchmark with Gemini 3.5 flash.

Key takeaway

For AI Engineers building knowledge agents, recognize that the "knowledge gap" between LLM reasoning and effective retrieval is a primary performance constraint. Your focus should shift towards designing sophisticated search agents that actively guide LLMs to generate precise semantic queries and intelligently leverage diverse retrieval tools. Consider fine-tuning smaller LLMs for agentic search to significantly enhance both performance and operational efficiency in complex knowledge work applications.

Key insights

LLMs' retrieval performance is bottlenecked by poor query generation, a gap addressable by specialized search agents.

Principles

• LLM reasoning capabilities outpace retrieval
• Query generation quality is a critical bottleneck
• Tailoring search tools improves agent performance

Method

Implement an agent harness with diverse search tools (semantic, keyword, filter), an iterative loop with parallel queries, and train via supervised fine-tuning and reinforcement learning using a combined retrieval and trajectory reward.

In practice

• Frame query writing as "one concise sentence"
• Provide initial corpus context to guide planning
• Use a small LLM for agent speed and cost

Topics

• LLM Retrieval
• Knowledge Agents
• Semantic Search
• Query Generation
• Reinforcement Learning
• Agentic AI

Best for: AI Architect, NLP Engineer, Research Scientist, AI Engineer, Machine Learning Engineer, AI Scientist

Related on AIssential

• Diagnosing Knowledge Gaps in LLM Tool Use: An Agentic Benchmark for Novel API Acquisition — LLM tool use with novel APIs requires specific knowledge components, with usage examples being critical, and retrieval/tuning playing complementary roles.
• Is Grep All You Need? How Agent Harnesses Reshape Agentic Search — Grep-based retrieval often outperforms vector retrieval in LLM agentic search, but performance varies by agent harness.
• Paper Digest: SIGIR 2026 Papers & Highlights — The information retrieval field is rapidly advancing, driven by LLMs, RAG, and multimodal approaches, with a strong focus on practical applications and robust evaluation.
• Building Search for AI Agents with Exa CEO Will Bryk — AI agents demand a new search paradigm focused on comprehensive, controllable, and deep information retrieval, distinct from human-optimized engines.
• A Late Chunking Approach for Visual Documents, Does Agentic Search Make GraphRAG Obsolete? and More! — New research enhances LLM retrieval, recommendation systems, and RAG through novel architectures and training methods.
• AI search agents don't fail at searching, they fail at asking the right questions when queries get ambiguous — AI search agents fail by not clarifying ambiguous queries, leading to error propagation.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by AI Engineer.