EvoBrowseComp: Benchmarking Search Agents on Evolving Knowledge

2026-06-11 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

EvoBrowseComp is a new, evolving benchmark designed to evaluate Search Agents, which are large language models augmented with search tools. Unlike existing benchmarks like BrowseComp that use static knowledge, EvoBrowseComp addresses vulnerabilities to test-set contamination and parametric memorization, where models rely on fact recall instead of genuine retrieval. This benchmark features 400 English and 400 Chinese complex questions, synthesized via live-web traversal using a three-agent collaborative framework. This framework includes a QA synthesis agent for fresh knowledge retrieval, an information filtering agent for credibility and popularity checks, and a high-level guidance agent to formalize questions into reasoning graphs. Its fully automated synthesis allows regular updates, preventing data contamination and ensuring temporal freshness, establishing a scalable paradigm for high-difficulty benchmarking.

Key takeaway

For Machine Learning Engineers developing or evaluating Search Agents, EvoBrowseComp offers a critical new tool. Your current benchmarks likely suffer from static knowledge and contamination, leading to inflated performance metrics. Adopting EvoBrowseComp will provide a more accurate assessment of genuine browsing competence by challenging models with evolving, contamination-free questions synthesized from the live web, ensuring your agent's capabilities are truly future-proof.

Key insights

EvoBrowseComp offers an evolving, contamination-free benchmark for Search Agents using a three-agent live-web synthesis framework.

Principles

• Benchmarks must evolve to counter contamination.
• Automated synthesis ensures temporal freshness.
• High-difficulty questions demand broad search.

Method

EvoBrowseComp synthesizes questions via a three-agent framework: a QA agent retrieves live-web knowledge, an information filtering agent checks credibility, and a guidance agent formalizes questions into reasoning graphs to reduce shortcuts.

Topics

• Search Agents
• LLM Benchmarking
• EvoBrowseComp
• Data Contamination
• Live-Web Data Synthesis
• Question Answering

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• SWE-Explore: Benchmarking How Coding Agents Explore Repositories — SWE-Explore benchmarks coding agents' repository exploration, revealing agentic methods surpass classical retrieval in line-level code localization.
• RiskWebWorld: A Realistic Interactive Benchmark for GUI Agents in E-commerce Risk Management — Foundational model scale significantly outperforms specialized GUI grounding in complex, high-stakes web automation tasks.
• Search-Time Contamination in Deep Research Agents: Measuring Performance Inflation in Public Benchmark Evaluation — Deep research agents' web search can contaminate benchmark evaluations, inflating performance by bypassing genuine reasoning.
• EvoMap / evolver — Evolver provides a protocol-constrained, auditable framework for AI agent self-evolution through GEP-guided prompt generation.
• LoHoSearch: Benchmarking Long-Horizon Search Agents Beyond the Human Difficulty Ceiling — Human-authored benchmarks limit complexity, necessitating automated knowledge graph-driven generation for challenging long-horizon search tasks.
• Evoflux: Inference-Time Evolution of Executable Tool Workflows for Compact Agents — Evoflux uses inference-time evolutionary search to repair executable tool workflows, significantly improving compact LM tool use.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.