What exactly is Speculative Decoding?

2026-07-13 · Source: LLM on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Advanced, long

Summary

Speculative Decoding is a technique that accelerates large language model (LLM) text generation by two to three times without altering the output tokens. Unlike other speed optimizations, it offers a "free lunch" by producing bit-for-bit identical text to what the large model would generate alone. The method employs a small, fast "draft" model to predict a burst of future tokens (e.g., gamma=4), which the large "target" model then verifies in a single parallel pass. This is efficient because a large model's forward pass is memory-bandwidth bound, meaning scoring multiple tokens costs almost the same as scoring one. If drafted tokens are correct, several words are accepted for the cost of one big-model step. The process is provably lossless, ensuring the emitted token distribution matches the target model's, with a bad draft only reducing speed, not correctness. Real-world speedups range from 2x to 3x, with TensorRT-LLM on H200 achieving up to 3.6x and EAGLE-3 up to 6.5x. This technique is most effective in low-latency, low-batch serving scenarios where the GPU has spare compute.

Key takeaway

For AI Engineers optimizing LLM inference latency, Speculative Decoding offers a significant speedup without compromising output quality. If you are deploying models in low-latency, low-batch serving environments, implementing this technique can yield 2x-3x faster generation. Be aware that it may cause slowdowns in high-batch, compute-bound scenarios, so carefully evaluate your operational context before adoption. Consider using a small draft model or exploring methods like Medusa or EAGLE for self-drafting.

Key insights

Speculative decoding uses a small draft model to predict tokens, verified by a large model in one pass, achieving lossless speedup.

Principles

• LLM generation is memory-bound, not compute-bound.
• Parallel verification of multiple tokens is nearly free.
• Output correctness is mathematically guaranteed.

Method

A small draft model proposes gamma tokens; the large target model verifies all in one parallel pass, accepting with `min(1, p/q)` and resampling rejections from `norm(max(0, p - q))`.

In practice

• Use a small model or n-gram for drafting.
• Apply in low-latency, low-batch serving.
• Avoid in compute-bound, high-batch scenarios.

Topics

• Speculative Decoding
• LLM Inference Optimization
• Autoregressive Generation
• Draft Models
• Medusa
• EAGLE-3

Best for: Machine Learning Engineer, AI Engineer, AI Scientist

Related on AIssential

• Speculative Decoding — Speculative decoding accelerates LLM inference by using a fast draft model to propose tokens and a large target model to verify them in parallel.
• Speculoos…No, Speculative Decoding: The Trick That Made My Old MacBook 3x Faster — Speculative decoding accelerates LLM inference by using a small draft model to predict tokens, verified by a larger model in parallel.
• The Next Frontier: How Speculative Decoding Is Eating the LLM Inference Stack — Speculative decoding boosts LLM inference speed by parallelizing token verification, leveraging idle GPU compute without quality loss.
• Your LLM Is Guessing Ahead. Then It Checks Itself aka Speculative Decoding — Speculative decoding accelerates LLM inference by using a small model to guess tokens, verified by a large model in one pass.
• Accelerating decode-heavy LLM inference with speculative decoding on AWS Trainium and vLLM — Speculative decoding accelerates LLM inference for predictable outputs by reducing sequential decode steps and improving hardware utilization.
• The Evolution of LLM Inference: Decoding algorithms — Part 1 — LLM inference speed hinges on reducing sequential decoding steps by predicting and verifying multiple tokens in parallel.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by LLM on Medium.