Metrics of Agent Ability

2026-07-24 · Source: METR · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Advanced, long

Summary

This post comprehensively reviews various metrics for evaluating AI agent capabilities, particularly when comparing agent performance against human benchmarks. It introduces a framework based on score functions $s_A(x)$ and $s_H(x)$, representing agent and human scores as a function of expenditure (e.g., money, tokens, time). The analysis categorizes metrics into agent-only approaches, such as "Score at Fixed Expenditure" and "Expenditure at Fixed Score," and human-grounded metrics like "Binary Time Horizon" and "Human-Relative Cost at Fixed Score." The author notes that human-grounded metrics enhance interpretability and generalizability, especially as agents achieve high performance. The discussion highlights the importance of accounting for test-time scaling and the complexities of measuring these expenditure-to-score curves accurately.

Key takeaway

For MLOps Engineers designing agent evaluation benchmarks, you must move beyond simple fixed-expenditure scores. Consider the full expenditure-to-score curve, $s(x)$, and integrate human-grounded metrics like "Human-Relative Cost at Fixed Score" to accurately assess agent capability and cost-efficiency. This approach ensures your evaluations remain relevant as AI agents achieve higher performance and exhibit significant test-time scaling, providing a clearer picture of true economic value.

Key insights

Agent capability metrics must account for expenditure and human baselines to be meaningful.

Principles

• If score curves are concave, raw score at fixed expenditure is sufficient.
• High returns to expenditure necessitate test-time scaling metrics.
• Benchmarking against human abilities improves task generalization.

Method

The article taxonomizes agent capability metrics based on agent-only or human-grounded comparisons, using expenditure-to-score curves $s_A(x)$ and $s_H(x)$ to define various evaluation approaches.

In practice

• Use $s_A(\bar{x})$ for fixed expenditure evaluations.
• Calculate $s_A^{-1}(\bar{s})$ for cost-efficiency.
• Employ $s_H^{-1}(s_A(\bar{x}_A))$ for human-equivalent cost.

Topics

• Agent Capability Metrics
• AI Evaluation
• Expenditure-to-Score Curves
• Human-Grounded Metrics
• Test-Time Scaling
• Benchmark Design

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• Do Agent Benchmarks Measure Capability? Protocol Validity in the Age of Agentic AI — Agent benchmarks often fail to measure true capability due to "reward hacking" and require rigorous protocol validity checks.
• 47 - David Rein on METR Time Horizons — AI's ability to complete long tasks, measured by human completion time, has increased exponentially over the past five years.
• WAC* — Standardized AI benchmarks fail to capture real-world utility, mirroring past human evaluation pitfalls.
• Your AI Agent Passed Every Eval. Finance Still Killed It. — AI agent evaluation must include "cost per successful outcome" to determine economic viability, not just quality.
• Are AI benchmarks doomed? — AI benchmarks, despite rapid saturation, remain vital for evaluating capabilities, with future development using AI and human judgment.
• Mastering Agentic Techniques: AI Agent Evaluation — AI agent evaluation assesses end-to-end system behavior in dynamic environments, distinct from static model capability benchmarks.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by METR.