Is Grep All You Need? How Agent Harnesses Reshape Agentic Search

2026-05-14 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Advanced, medium

Summary

An empirical study investigated the performance of Large Language Model (LLM) agents using different retrieval methods and tool-calling styles. The research, detailed in "Is Grep All You Need? How Agent Harnesses Reshape Agentic Search," involved two experiments. Experiment 1 compared `grep` and vector retrieval on 116 questions from LongMemEval, utilizing a custom agent harness (Chronos) and provider-native CLIs (Claude Code, Codex, Gemini CLI), assessing both inline and file-based tool results. Experiment 2 focused on `grep`-only versus vector-only retrieval, progressively introducing unrelated conversation history to evaluate robustness against distracting material. The findings indicate that `grep` generally achieves higher accuracy than vector retrieval across Chronos and provider CLIs in Experiment 1, though overall scores are significantly influenced by the specific harness and tool-calling approach used.

Key takeaway

For AI Architects and Research Scientists designing LLM agentic workflows, you should prioritize `grep`-based retrieval over vector retrieval for initial implementations, especially when accuracy is paramount. Your choice of agent harness and how tool outputs are presented significantly impacts performance, even with identical underlying data. Consider robust testing across different harnesses and tool output formats to optimize your agent's effectiveness and resilience against irrelevant context.

Key insights

`grep` often outperforms vector retrieval in LLM agentic search, but harness and tool-calling style are critical.

Principles

• Tool output presentation impacts agent performance.
• Irrelevant context degrades search accuracy.
• Harness choice strongly influences agent scores.

Method

The study used two experiments: one comparing `grep` and vector retrieval across multiple harnesses and tool-calling styles, and another assessing retrieval robustness against increasing irrelevant conversational noise.

In practice

• Prioritize `grep` for initial agentic search implementations.
• Carefully select agent harnesses for optimal performance.
• Design tool output presentation for clarity.

Topics

• LLM Agents
• Agentic Search
• Grep Retrieval
• Vector Retrieval
• Agent Harnesses

Code references

• 8421BCD/Agentic-R
• o2lab/afc-crs-all-you-need-is-a-fuzzing-brain
• EmergenceAI/Agent-E

Best for: AI Architect, Research Scientist, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• Is Grep All You Need? How Agent Harnesses Reshape Agentic Search — Grep-based retrieval often outperforms vector retrieval in LLM agentic search, but performance varies by agent harness.
• Researchers trained an open source AI search agent, Harness-1, that outperforms GPT-5.4 on recalling relevant information — Externalizing search state into a structured environment drastically improves AI agent performance and data efficiency.
• Bridging intent and execution in agentic systems — Minimizing the intent-execution gap in agent harnesses is crucial for state-of-the-art LLM performance.
• ScholarQuest: A Taxonomy-Guided Benchmark for Agentic Academic Paper Search in Open Literature Environments — ScholarQuest benchmark reveals LLM-based search agents significantly underperform in academic literature exploration, despite outperforming baselines.
• not much happened today — AI agent tooling, infrastructure, and robotics are rapidly advancing, while platform stability and search access pose new challenges.
• SWE-Explore: Benchmarking How Coding Agents Explore Repositories — SWE-Explore benchmarks coding agents' line-level repository exploration, isolating it from end-to-end repair outcomes.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.