MASEval: Extending Multi-Agent Evaluation from Models to Systems

2026-02-07 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Robotics & Autonomous Systems · Depth: Advanced, extended

Summary

MASEval is a new framework-agnostic evaluation library designed for multi-agent LLM-based systems, addressing a gap in existing model-centric benchmarks. Developed by Parameter Lab, University of Oxford, and others, MASEval treats the entire agent system, including topology, orchestration logic, and error handling, as the unit of analysis. Through systematic comparisons across 3 benchmarks, 3 models (GPT-5-mini, Gemini-3.0-Flash, Claude-Haiku-4.5), and 3 frameworks (smolagents, LangGraph, LlamaIndex), MASEval found that framework choice impacts performance comparably to model choice within the same capability tier. For instance, the mean performance range across models was 14.2 percentage points (pp), while across frameworks it was 12.4 pp. The library also significantly reduces implementation effort for benchmark consumers (83–91%) and producers (35–57%). MASEval is open-source under the MIT license and available at github.com/parameterlab/MASEval.

Key takeaway

For AI Engineers and Research Scientists building multi-agent systems, your choice of framework is as critical as your choice of LLM. You should utilize system-level evaluation tools like MASEval to systematically compare framework implementations and architectural decisions, rather than relying solely on model-centric benchmarks. This approach will help you identify optimal configurations and avoid performance pitfalls arising from framework-model interactions, leading to more robust and efficient agent deployments.

Key insights

Framework choice significantly impacts multi-agent system performance, often as much as model choice within a capability tier.

Principles

• Evaluate the complete agent system, not just models.
• Prioritize framework-agnostic, "Bring Your Own" evaluation.
• Separate concerns: task, environment, agent logic, evaluation.

Method

MASEval provides a unified evaluation layer with abstract base classes for agents, environments, and evaluators, orchestrating a five-phase benchmark lifecycle (Setup, Execute, Collect, Evaluate, Report) with multi-agent tracing.

In practice

• Use MASEval to compare multi-agent framework implementations.
• Implement custom adapters for new agent frameworks.
• Leverage adaptive testing to reduce LLM evaluation costs.

Topics

• Multi-Agent Systems
• LLM Evaluation
• Agent Frameworks
• System-Level Benchmarking
• Performance Analysis

Code references

• parameterlab/MASEval
• anthropics/claude-agent-sdk-python
• Arize-ai/phoenix
• braintrustdata/braintrust-sdk
• open-compass/opencompass

Best for: AI Engineer, AI Scientist, Research Scientist, AI Researcher, Machine Learning Engineer, AI Architect

Related on AIssential

• Do More Agents Help? Controlled and Protocol-Aligned Evaluation of LLM Agent Workflows — Multi-agent LLM workflows do not consistently outperform single-agent setups under controlled evaluation, except for advanced runtime-generated systems.
• AI Agent Evaluation: How to Know If Your Agent Actually Works — AI agent evaluation requires a system-level approach, measuring task completion, cost, and latency across layered test suites and continuous integration.
• The Illusion of Multi-Agent Advantage — Automated Multi-Agent Systems often incur high costs for superficial complexity, failing to outperform simpler Single-Agent Systems.
• Benchmark Everything Everywhere All at Once — Autonomous agentic systems can significantly automate and scale benchmark creation for LLMs and MLLMs.
• Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents — Agentic CLEAR automates multi-level evaluation for LLM agents, providing dynamic, data-driven insights into their behavior.
• AI Agents of the Week: Papers You Should Know About — Modular multi-agent systems with enhanced tool use and memory can surpass monolithic LLMs in complex, real-world tasks.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.