A new GPT-OSS benchmark and DeepSeek R1 updates for latency-optimized reasoning

2026-03-24 · Source: MLCommons · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Advanced, medium

Summary

The MLPerf Inference v6.0 release significantly expands its coverage of open-weight large language models, introducing two major additions. First, the GPT-OSS 120B benchmark features a 117B total parameter Mixture-of-Experts (MoE) model (5.1B active per token) designed for mathematics, scientific reasoning, and coding. This benchmark uniquely separates performance (low-effort summarization, 10,240 tokens max output) and accuracy datasets (high-effort tasks like AIME 2024, LiveCodeBench v6, GPQA-Diamond, 32,768 tokens max output), with specific accuracy targets of 82.92%, 74.95%, and 84.68% respectively. Second, a new DeepSeek-R1 interactive scenario adds a low-latency workload for real-time reasoning, featuring 99th percentile TTFT <= 1.5s and TPOT <= 15ms. This scenario also introduces the first standard for speculative decoding in MLPerf, specifically using EAGLE-style decoding with the DeepSeek-R1 MTP Head.

Key takeaway

For MLOps Engineers deploying open-weight LLMs, MLPerf Inference v6.0 provides critical new benchmarks. You should now evaluate models like GPT-OSS 120B using separate performance and accuracy datasets to reflect diverse production workloads. For real-time reasoning applications, implement speculative decoding with the DeepSeek-R1 MTP Head to meet strict latency targets, ensuring your deployments remain competitive and efficient.

Key insights

MLPerf Inference v6.0 introduces new benchmarks for open LLMs, separating performance and accuracy, and standardizing speculative decoding.

Principles

• Benchmarks must reflect evolving LLM architectures.
• Separate datasets optimize performance and accuracy evaluation.
• Low-latency scenarios require specific decoding standards.

Method

MLPerf v6.0 defines a split-dataset strategy for GPT-OSS, using distinct datasets for performance (summarization) and accuracy (complex reasoning), and mandates EAGLE-style speculative decoding for DeepSeek-R1 interactive.

In practice

• Evaluate LLMs with split performance/accuracy datasets.
• Implement EAGLE-style speculative decoding for low latency.
• Use OpenAI Harmony chat format for reasoning control.

Topics

• MLPerf Inference
• Large Language Models
• GPT-OSS 120B
• DeepSeek-R1
• Speculative Decoding
• LLM Benchmarking
• Mixture-of-Experts

Code references

• openai/gpt-oss
• v-shobhit/gpt-oss
• mlcommons/inference
• mlcommons/inference_policies

Best for: AI Engineer, NLP Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• **Introducing SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding** — SPEED-Bench offers a unified benchmark for Speculative Decoding, addressing real-world data diversity and serving conditions.
• Reproducing the AMD MLPerf Inference v6.0 Submission Result — AMD provides a guide to reproduce its MLPerf Inference v6.0 results on Instinct MI355X platforms.
• DeepSeek-V3.2 Outperforms GPT-5 on Reasoning Tasks — DeepSeek-V3.2 offers open-source reasoning and agentic AI models, outperforming GPT-5 on benchmarks via novel attention and scaled RL.
• 2026 Predictions: Much Faster Inference, Pre-Training with RL, and FP4 Everywhere — DeepSeek's MoE and RLVR advancements, alongside FP4 quantization, defined 2025 LLM progress, while Llama 4 disappointed.
• DeepSeek-V3: A Large-Scale MoE Pretraining Benchmark for MLPerf Training v6.0 — MLPerf Training v6.0 now benchmarks MoE LLMs, reflecting industry shifts to sparse computation and specialized architectures.
• NVIDIA Extreme Co-Design Delivers New MLPerf Inference Records — Co-designed hardware, software, and models are crucial for maximizing AI factory throughput and minimizing token cost.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by MLCommons.