Evals are the modern PRD—they define "what" success looks like, not "how" to achieve it

2026-06-18 · Source: How I AI · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Project & Product Management · Depth: Intermediate, quick

Summary

Artificial intelligence, particularly machine learning and transformer models, fundamentally alters the programming paradigm from specifying "how" a task is accomplished to defining "what" the desired outcome or success looks like. This shift is analogous to statistical regression, where users provide data points (the "what"), and the system computes the slope and Y-intercept (the "how"). For modern AI, such as transformers, the "what" is the task of predicting the next token, while the underlying computational substrate and GPUs determine "how" to achieve that prediction. This reorientation towards defining outcomes, rather than explicit instructions, is presented as a key to increased productivity in AI development, positioning evaluations (Evals) as the modern Product Requirements Document (PRD) for defining success.

Key takeaway

For AI Product Managers defining project scope, recognize that AI development thrives on clearly articulated outcomes. Shift your focus from detailing implementation "how" to precisely defining "what" success means for your models and applications. This means prioritizing robust evaluation metrics (Evals) as your primary specification, enabling your teams to innovate on the execution while ensuring alignment with business goals.

Key insights

AI shifts programming from defining "how" to achieve tasks to specifying "what" success looks like, enhancing productivity.

Principles

• AI development prioritizes outcome definition over procedural instruction.
• Modern AI systems determine execution "how" from defined "what."
• Focusing on "what" (outcomes) boosts AI productivity.

In practice

• Define success metrics (Evals) before implementation.
• Focus on desired outputs for AI systems.
• Frame AI problems as "what" to predict or achieve.

Topics

• AI Programming Paradigm
• Machine Learning
• Transformers
• Next Token Prediction
• Product Requirements Document
• AI Evaluation

Best for: AI Product Manager, AI Architect, Director of AI/ML

Related on AIssential

• Evals Skills for Coding Agents — Improving infrastructure around AI agents, especially evaluation capabilities, is more critical than solely improving the underlying model.
• Quoting Boris Mann — Quantifying AI agents without context is as uninformative as counting spreadsheets or browser tabs.
• The AI Progress Chart Everyone Is Misreading — Beth Barnes & David Rein — AI evaluation requires unified metrics like human task completion time to track progress and understand capabilities across diverse models.
• State of the art LLMs — AI agents can satirically replicate human organizational inefficiencies, leading to high costs with no tangible output.
• Building an Evaluation Harness for Production AI Agents: A 12-Metric Framework From 100+ Deployments — Robust evaluation infrastructure is critical for successful, compliant AI agent deployments in production.
• The maturity phases of running evals — Phil Hetzel, Braintrust — Evals mature from human judgment to automated, production-integrated systems, crucial for agent quality and risk management.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by How I AI.