Agentic Inference for MLPerf Inference

2026-07-08 · Source: MLCommons · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Advanced, medium

Summary

The MLPerf Agentic Inference benchmark has been introduced to address the growing complexity of multi-turn agentic large language model applications, such as coding assistants and workflow agents. This new benchmark extends the MLPerf Endpoints framework to measure performance for "trajectories," which are sequences of dependent turns where context grows over time, increasing KV-cache pressure and requiring efficient reuse. It evaluates two models, Kimi K2.6 (1T/32B active parameters, 262,144 tokens context) and Qwen3.6-35B-A3B (35B/3B active parameters, 262,144 tokens context), using a dataset of 990 multi-turn trajectories. These include 490 agentic coding traces, stressing KV-cache capacity, and 500 agentic workflow traces, emphasizing shared-prefix reuse. Performance is assessed via a Pareto curve, balancing aggregate throughput and per-user progress, with accuracy enforced through OSL mean value, inline accuracy, and standalone accuracy checks.

Key takeaway

For MLOps Engineers building serving infrastructure for multi-turn LLM agents, traditional single-turn benchmarks are insufficient. You should adopt the new MLPerf Agentic Inference benchmark to accurately assess system performance. Focus your optimizations on preserving KV-cache locality, efficiently scheduling long contexts, and balancing aggregate throughput with individual agent progress, as these are critical for real-world agentic application success. This benchmark provides a standardized way to validate your system's readiness for complex, conversational AI workloads.

Key insights

MLPerf Agentic Inference benchmarks multi-turn LLM workloads, focusing on context growth, KV-cache, and turn dependencies.

Principles

• Agentic inference demands benchmarks for growing context.
• KV-cache reuse is critical for multi-turn LLM efficiency.
• Throughput must measure closed-loop progress.

Method

The Agentic Inference benchmark measures multi-turn LLM performance by driving OpenAI-compatible endpoints with closed-loop replay of 990 coding and workflow trajectories, validating accuracy at three levels.

In practice

• Use X-Session-ID for KV-cache locality.
• Implement prefix caching and speculative decoding.
• Balance aggregate throughput with per-user progress.

Topics

• Agentic Inference
• MLPerf Benchmarks
• LLM Serving
• KV-cache Optimization
• Multi-turn AI
• Speculative Decoding

Code references

• mlcommons/endpoints

Best for: AI Engineer, MLOps Engineer, Machine Learning Engineer, AI Architect

Related on AIssential

• Benchmark Everything Everywhere All at Once — Autonomous agentic systems can significantly automate and scale benchmark creation for LLMs and MLLMs.
• A new GPT-OSS benchmark and DeepSeek R1 updates for latency-optimized reasoning — MLPerf Inference v6.0 introduces new benchmarks for open LLMs, separating performance and accuracy, and standardizing speculative decoding.
• LMCache / LMCache — LMCache transforms KV cache into reusable, persistent "AI-native knowledge" to boost LLM inference performance.
• NVIDIA Achieves Leading Agentic Coding Performance on First Agentic AI Benchmark — AA-AgentPerf defines the first standard for measuring AI agentic coding performance, revealing significant hardware efficiency gains.
• not much happened today — AI agent development is shifting from interactive chat to robust, verifiable, and automated production systems.
• Inference Optimization in LLMs: A Systems View — Inference optimization for LLMs is a layered systems problem, not a single technique, where memory bandwidth is the primary constraint.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by MLCommons.