Stop Ranking Agent Configs by Average Score

2026-07-06 · Source: Towards Data Science · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Advanced, long

Summary

The article introduces a method for evaluating agent configurations that moves beyond simple average scoring, advocating for best-worst scaling (MaxDiff) combined with a Plackett-Luce model. This approach aims to uncover interaction effects between agent components like models, prompts, and tools. An experiment on an invoice extraction agent, utilizing the "shubh303/Invoice-to-Json" dataset, compared 8 configurations across three binary factors: Claude Haiku 4.5 vs. GPT-5.4-mini models, Systematic Planner vs. Contextual Leaper prompts, and stream table rows vs. semantic search tools. A judge model, `claude-sonnet-4-6`, selected the best and worst outputs from batches of 5 configurations. The analysis revealed that a three-way interaction between model, prompt, and tool was the most significant factor (0.31 importance), indicating that component combinations often matter more than individual strengths.

Key takeaway

For AI Engineers evaluating agent pipelines, relying solely on average scores for configuration selection risks deploying suboptimal systems. You should implement best-worst scaling with a ranking model like Plackett-Luce to identify critical interaction effects between models, prompts, and tools. This approach provides a more accurate utility map, enabling you to deploy robust configurations that perform well as a unit, even if individual components appear weaker in isolation.

Key insights

Agent evaluation should use direct best-worst comparisons and ranking models to uncover component interaction effects.

Principles

• Agent performance is often interactional, not additive.
• Direct comparison reveals more signal than isolated scores.
• Weaker components can form strong configurations.

Method

Run competing agent configurations on shared examples, ask a judge to select best/worst outputs, then fit a Plackett-Luce or MaxDiff model to estimate utility scores.

In practice

• Use a judge agent to automate best-worst comparisons.
• Code factor levels as +1/-1 for main and interaction effects.
• Feed estimated utilities into agent routing context.

Topics

• Agent Evaluation
• Plackett-Luce Model
• Best-Worst Scaling
• LLM Agents
• Configuration Optimization
• Interaction Effects
• Invoice Extraction

Code references

• doc1000/agent-config-optimization

Best for: AI Engineer, MLOps Engineer, Director of AI/ML

Related on AIssential

• Towards Pareto-Optimal Tool-Integrated Agents with Pareto Ranking Policy Optimization — ParetoPO optimizes tool-integrated LLMs for superior accuracy-efficiency trade-offs using a two-stage multi-objective framework.
• GrowthHacker: Automated Off-Policy Evaluation Optimization Using Code-Modifying LLM Agents — LLM-based agents can autonomously optimize Off-Policy Evaluation code, significantly improving performance and reliability in data-driven systems.
• Popular LLM ranking platforms are statistically fragile, new study warns — Popular LLM ranking platforms exhibit high statistical fragility, with minimal user feedback changes altering top model rankings.
• Smarter Saboteurs, Better Fixers: Scaling & Security in Linear Multi-Agent Workflows — Larger LLMs in multi-agent systems are more vulnerable to attacks, but a simple "Fixer" stage restores resilience.
• Picking the right model and agent — at the right cost and latency — shouldn’t be a guess — SmartWrapperOSS enables objective comparison of LLM performance and cost for agentic and summarization tasks, moving beyond guesswork.
• Introducing the agent quality loop: AgentCore Optimization now in preview — AgentCore Optimization automates AI agent improvement through data-driven recommendations and rigorous validation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Towards Data Science.