NVIDIA Extreme Co-Design Delivers New MLPerf Inference Records

2026-04-01 · Source: NVIDIA Technical Blog · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Cloud Computing & IT Infrastructure, Software Development & Engineering · Depth: Advanced, medium

Summary

NVIDIA's Blackwell Ultra GPUs achieved the highest throughput across the widest range of models and scenarios in the MLPerf Inference v6.0 benchmarks, bringing their cumulative MLPerf training and inference wins since 2018 to 291. This round introduced new tests for models like DeepSeek-R1 Interactive, Qwen3-VL-235B-A22B (the first multi-modal model), GPT-OSS-120B, WAN-2.2-T2V-A14B (text-to-video), and DLRMv3 (generative recommendation). NVIDIA was the only platform to submit results on all new models and scenarios, demonstrating up to 2.7x performance gains on DeepSeek-R1 and 1.5x on Llama 3.1 405B on GB300 NVL72 systems, attributed to software optimizations like TensorRT-LLM updates and scale-out inference with Quantum-X800 InfiniBand.

Key takeaway

For AI engineers and CTOs evaluating infrastructure for large-scale AI deployments, NVIDIA's MLPerf Inference v6.0 results indicate that their full-stack approach, combining Blackwell Ultra GPUs with optimized software like TensorRT-LLM and Quantum-X800 InfiniBand, delivers industry-leading throughput and cost efficiency. You should consider NVIDIA's integrated platform for demanding generative AI, multimodal, and recommendation workloads to maximize token output and reduce operational costs.

Key insights

Co-designed hardware, software, and models are crucial for maximizing AI factory throughput and minimizing token cost.

Principles

• Rigorous benchmarks are essential for real-world AI inference performance.
• Continuous software optimization improves existing hardware performance.
• Scale-out networking enables massive token processing rates.

Method

NVIDIA achieved performance gains through faster kernels, Optimized Attention Data Parallel, disaggregated serving, Wide Expert Parallel (WideEP), Multi-Token Prediction (MTP), and KV-aware routing.

In practice

• Utilize TensorRT-LLM for LLM inference serving.
• Employ NVIDIA Dynamo for distributed inference serving.
• Integrate Quantum-X800 InfiniBand for scale-out inference.

Topics

• MLPerf Inference v6.0
• NVIDIA Blackwell Ultra GPUs
• TensorRT-LLM
• DeepSeek-R1
• Multi-modal AI Models

Code references

• NVIDIA/TensorRT-LLM
• NVIDIA/recsys-examples
• NVIDIA/nv-embedding-cache

Best for: AI Engineer, CTO, VP of Engineering/Data, Machine Learning Engineer, MLOps Engineer, AI Architect

Related on AIssential

• How NVIDIA Extreme Hardware-Software Co-Design Delivered a Large Inference Boost for Sarvam AI’s Sovereign Models — Full-stack co-optimization of model design, kernels, scheduling, and hardware significantly boosts LLM inference performance.
• NVIDIA Blackwell Tops MLPerf Training 6.0 with Industry-Leading Scale and Performance — NVIDIA's Blackwell platform achieved MLPerf Training v6.0 dominance through full-stack hardware-software co-design and advanced optimizations.
• A new GPT-OSS benchmark and DeepSeek R1 updates for latency-optimized reasoning — MLPerf Inference v6.0 introduces new benchmarks for open LLMs, separating performance and accuracy, and standardizing speculative decoding.
• Building a monokernel for LLM inference on AMD MI300X - up to 3,300 output tokens/s per request [P] — Optimizing LLM inference on AMD MI300X by leveraging hardware die topology with a single GPU-resident monokernel.
• Lambda's MLPerf Inference v6.0: hardware leap, software maturity, research breakthrough — Hardware, software, and routing optimizations significantly boost AI inference performance and reduce latency.
• Leading Inference Providers Cut AI Costs by up to 10x With Open Source Models on NVIDIA Blackwell — Extreme co-design across the entire data center stack is crucial for cost-effective, high-performance AI reasoning.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by NVIDIA Technical Blog.