It's 2026, and We're Still Talking Evals

2026-04-20 · Source: MLOps.community · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics, Software Development & Engineering · Depth: Advanced, quick

Summary

Maggie Konstanty, an AI Product Manager at Prosus, discusses the complexities of evaluating AI agents, particularly for large-scale food ordering and e-commerce. She highlights that simple accuracy metrics like "95% accurate" are insufficient for real-world performance, advocating for a shift from pre-ship to production-focused evaluations. Konstanty emphasizes the importance of user drop-off analytics as a critical, yet underutilized, signal for agent failure. She critiques the "20-evaluator trap," stressing the need to design evaluations directly tied to product goals and real user behavior, citing a "surprise me" edge case from Prosus's food ordering agent. Furthermore, she expresses skepticism about LLM-as-a-judge for accuracy in production, preferring alternative approaches, and offers a candid assessment of current eval platforms like Arize/Phoenix, noting that mature teams often revert to custom code. Konstanty concludes by advocating for evaluations as an embedded team practice and suggests using incentive-based red teaming.

Key takeaway

For AI Product Managers and ML engineers struggling to make evaluations meaningful, recognize that pre-ship metrics often fail in production. Focus your evaluation strategy on real user behavior, leveraging drop-off analytics and designing tests directly linked to product outcomes. Consider implementing incentive-based red teaming to proactively identify agent weaknesses and foster a culture where evaluation is a continuous, integrated team practice, rather than a one-time checklist item.

Key insights

Effective AI agent evaluation requires moving beyond simple accuracy to real-world user behavior and production-aligned metrics.

Principles

• User drop-off is a critical failure signal.
• Evals must align with product goals.
• Evaluation is a continuous team practice.

Method

Design evaluations tied to real outcomes, prioritize user drop-off analytics, and consider incentive-based red teaming to identify agent vulnerabilities.

In practice

• Monitor user drop-off rates closely.
• Tie eval metrics to business KPIs.
• Implement red teaming with incentives.

Topics

• LLM Evaluation
• AI Agent Performance
• Production Evaluation
• User Drop-off Analytics
• Evaluation Tooling

Best for: AI Product Manager, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• AI Dev 26 x SF | Ara Khan: Evals Are Broken Use Them Anyway — Evals are imperfect but essential for AI agent development, requiring nuanced interpretation and iterative improvement.
• Building production-ready LLM-powered applications @ Scale By the Bay — Transitioning LLM prototypes to production requires a pragmatic, product-focused approach centered on continuous evaluation and user outcomes.
• AI Agent Evaluation: Building Reliable Systems Beyond Simple Testing — Agent evaluation must assess multi-step processes, not just final outputs, due to compounding errors and non-determinism.
• The Sequence Opinion #860: Every Company’s Last eXam: Some Reflection About Practical AI Evals — Company-specific, dynamic evaluations are now essential for production AI, forming a fourth pillar alongside compute, data, and models.
• Online Evals Done Right: Runtime Scoring and Review Queues for Production LLM Systems — Effective online LLM evaluation prioritizes deterministic checks and structured escalation over early tooling adoption.
• Braintrust CEO: Evals are the new PRD for AI products — Evals serve as the modern PRD for AI products, enabling coding agents to rigorously test and optimize complex engineering challenges.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by MLOps.community.