**Introducing SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding**

2026-03-19 · Source: Hugging Face - Blog · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Intermediate, long

Summary

NVIDIA introduced SPEED-Bench on March 19, 2026, a unified benchmark for evaluating Speculative Decoding (SD) in Large Language Models (LLMs). SD accelerates LLM inference by using a lightweight draft model to speculate future tokens, which are then verified by the target model, improving throughput while preserving output distribution. Existing SD benchmarks are often fragmented and unrepresentative of real-world data and serving conditions, failing to account for data-dependent, serving-regime-dependent, and system-dependent factors. SPEED-Bench addresses these gaps with two dataset splits: a "Qualitative" split with 880 prompts across 11 diverse semantic categories to measure draft accuracy (conditional acceptance rates and lengths), and a "Throughput" split with 1,536 prompts per Input Sequence Length (ISL) bucket (1k to 32k tokens) to evaluate system-level speedups under high concurrency. It also includes a unified measurement framework integrated with production inference engines like TensorRT-LLM, vLLM, and SGLang, ensuring consistent evaluation by handling tokenization and prompt formatting externally.

Key takeaway

For AI Engineers and Research Scientists optimizing LLM inference, SPEED-Bench provides a robust framework to evaluate Speculative Decoding performance. You should integrate SPEED-Bench into your evaluation workflows to accurately assess draft model quality across diverse semantic domains and measure system-level speedups under realistic production serving conditions, avoiding the pitfalls of less representative benchmarks like those using random token inputs.

Key insights

SPEED-Bench offers a unified benchmark for Speculative Decoding, addressing real-world data diversity and serving conditions.

Principles

• SD performance is data, serving-regime, and system-dependent.
• Semantic diversity is critical for exposing domain-dependent SD behavior.
• Random tokens distort SD and MoE throughput measurements.

Method

SPEED-Bench uses two data splits (Qualitative for semantic diversity, Throughput for system speedups) and a unified measurement framework that pre-tokenizes inputs for consistent evaluation across production inference engines.

In practice

• Evaluate SD draft accuracy across diverse semantic domains.
• Measure system-level speedups under varying batch sizes and ISLs.
• Avoid random token inputs for SD and MoE throughput benchmarks.

Topics

• Speculative Decoding
• LLM Inference Benchmarking
• Semantic Diversity
• Inference Throughput
• Production Inference Engines

Code references

• NVIDIA/Model-Optimizer

Best for: AI Scientist, Research Scientist, AI Engineer, AI Researcher, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• Speculoos…No, Speculative Decoding: The Trick That Made My Old MacBook 3x Faster — Speculative decoding accelerates LLM inference by using a small draft model to predict tokens, verified by a larger model in parallel.
• A new GPT-OSS benchmark and DeepSeek R1 updates for latency-optimized reasoning — MLPerf Inference v6.0 introduces new benchmarks for open LLMs, separating performance and accuracy, and standardizing speculative decoding.
• DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verification — Dynamic ensemble verification in speculative decoding significantly boosts token acceptance and inference speed without sacrificing quality.
• An Interpretable Latency Model for Speculative Decoding in LLM Serving — A new latency model explains speculative decoding performance in LLM serving under dynamic loads.
• Your LLM Is Guessing Ahead. Then It Checks Itself aka Speculative Decoding — Speculative decoding accelerates LLM inference by using a small model to guess tokens, verified by a large model in one pass.
• 8x Faster Inference: DFlash with Block Diffusion Draft Trees — DDTree and IceCache offer distinct approaches to enhance LLM inference efficiency through improved speculative decoding and KV-cache management.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Hugging Face - Blog.