WebRetriever: A Large-Scale Comprehensive Benchmark for Efficient Web Agent Evaluation
Summary
WebRetriever is a new large-scale benchmark designed to address limitations in existing web agent evaluation frameworks, which suffer from insufficient scale, limited domain diversity, and inadequate assessment of real-world deployment needs. It comprises 1,550 tasks across 800 websites, spanning consumer, professional, and enterprise sectors with comprehensive user intent patterns. Complementing this, NavEval is a novel LLM-as-Judge framework that utilizes rich interaction context beyond visual screenshots, achieving over 90% alignment with human judgment. The benchmark introduces three evaluation protocols: navigation proficiency, knowledge-assisted interaction, and end-to-end task completion with information extraction. Experimental analysis reveals significant performance disparities, with agents achieving only 21.1% success rate for basic navigation (Protocol I) and 11.8% for end-to-end tasks (Protocol III). Operational documentation improved success rates to 29.2% on average. WebRetriever provides fine-grained diagnostic insights, highlighting current agents' inability to reliably handle practical end-to-end web tasks.
Key takeaway
For AI Engineers evaluating web agents for real-world deployment, you must move beyond basic navigation success metrics. Your evaluation framework should incorporate diverse, large-scale benchmarks like WebRetriever, covering varied domains and user intents. Crucially, assess end-to-end task completion, including information extraction, and integrate operational knowledge to improve agent reliability. Rely on robust automated evaluation methods like NavEval, achieving over 90% human agreement, for fine-grained diagnostic insights.
Key insights
Web agent evaluation requires large-scale, diverse benchmarks and fine-grained, multi-modal LLM-as-Judge methods to reflect real-world performance.
Principles
- Navigation success alone is an insufficient metric.
- External knowledge significantly improves agent task completion.
- End-to-end task completion with information extraction is critical.
Method
NavEval integrates task descriptions, URLs, filtered web requests, executed actions, and final screenshots, applying rule-based filtering to enhance LLM-as-Judge accuracy.
In practice
- Implement multi-protocol evaluation for web agents.
- Provide operational documentation to agents for complex tasks.
- Rigorously assess agent information extraction capabilities.
Topics
- Web Agents
- Benchmark Evaluation
- WebRetriever
- NavEval
- LLM-as-Judge
- Information Extraction
- Operational Documentation
Code references
Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Engineer
Related on AIssential
Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.