Benchmarks in Microsoft Foundry (preview): Standardized model and agent quality checks

2026-06-15 · Source: Microsoft Foundry Blog articles · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure · Depth: Intermediate, medium

Summary

Microsoft Foundry (preview) introduces Benchmarks, a new feature enabling standardized quality checks for AI model deployments and agents. This capability allows users to run well-known open-source benchmarks, such as AIME 2025 (30 examples) and BBEH (4,520 examples), directly against their specific configurations. Unlike the general model leaderboard, Benchmarks provide "right now" performance insights for custom deployments, fine-tunes, or agents, accounting for judge models and configurations. Each benchmark includes a curated dataset, task category (e.g., reasoning, math, truthfulness), and evaluation logic, with an optional judge model for scoring. Users can compare runs side-by-side in the evaluation group view, tracking scores (e.g., 82% with 645 / 790 examples passing) and token usage, accessible via the portal or REST API. This helps identify regressions after model upgrades or agent changes.

Key takeaway

For MLOps Engineers managing AI deployments and agents, Microsoft Foundry's Benchmarks feature provides a critical tool for validating performance and preventing regressions. You should integrate these standardized benchmarks into your CI/CD pipeline to automatically measure the impact of model upgrades, fine-tunes, or agent prompt/tool changes. This ensures consistent, reproducible quality checks, allowing you to quickly assess if a change improved or degraded your system's reasoning, math, or truthfulness capabilities before deployment.

Key insights

Microsoft Foundry Benchmarks standardize AI model and agent evaluation by integrating open-source benchmarks into the development workflow.

Principles

• Measure changes on the same benchmark, same conditions.
• Consistency in judge model is crucial for comparison.
• Token usage is a key metric alongside quality scores.

Method

Create an evaluation group, select target model/agent deployments, choose predefined benchmarks (and judge model if needed), then submit to get side-by-side scores and token usage.

In practice

• Compare two model deployments for quality vs. cost.
• Validate fine-tunes or check agent regressions.
• Use reasoning benchmarks for agent changes.

Topics

• Microsoft Foundry
• AI Model Evaluation
• Agent Benchmarking
• Performance Metrics
• REST API
• Quality Assurance

Best for: AI Architect, NLP Engineer, MLOps Engineer, Machine Learning Engineer, AI Engineer

Related on AIssential

• Evaluating AI Agents: A Practical Guide with Microsoft Foundry — Early and continuous evaluation is crucial for AI agents to catch regressions and ensure quality.
• Top 10: AI Benchmarking Tools — AI benchmarking is critical for validating performance, safety, and compliance across increasingly complex models and hardware.
• Why Do Performance Benchmarks Matter? — Benchmarks provide a standardized, real-world performance preview for AI platform evaluation.
• Microsoft Foundry: An End-to-End Platform for Building, Governing, and Scaling AI — Microsoft Foundry unifies AI development and operations for secure, scalable enterprise AI adoption.
• Introducing the agent quality loop: AgentCore Optimization now in preview — AgentCore Optimization automates AI agent improvement through data-driven recommendations and rigorous validation.
• We’ve Been Optimizing for the Wrong Thing: What Stanford’s Latest AI Paper Reveals About True… — AI benchmarks reward pattern recognition, not true reasoning, leading to "application-specific failure" in real-world tasks.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Microsoft Foundry Blog articles.