Windowed-MTP: Removing the Full-Context Draft-KV Tax at Million-Token Context

2026-04-07 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, extended

Summary

Windowed-MTP is a novel, training-free technique designed to eliminate the "long-context draft tax" in speculative decoding for large language models. This method applies a StreamingLLM-style sliding window and attention sink exclusively to the draft's attention mechanism, leaving the full-attention target verification intact. Benchmarked on Qwen GDN-MoE 35B/122B and Mamba2-hybrid NoPE 120B models at 1M context on a single GPU, Windowed-MTP reduces the per-decode-step cost by +28% to +44%. It also reclaims 7.7-11% of the total KV cache memory, improving throughput and latency. The approach is lossless, preserving the target's output distribution, and its benefits scale with context length, making deep speculation viable.

Key takeaway

For Machine Learning Engineers optimizing LLM inference at million-token contexts, the "long-context draft tax" on speculative decoding's MTP heads is a critical bottleneck. You should implement Windowed-MTP to cut per-decode-step costs by 28-44% and reclaim significant KV cache memory. This training-free, lossless technique ensures your long-context LLM deployments achieve higher throughput and lower latency without compromising output quality.

Key insights

Windowing the speculative decoding draft's attention at long contexts significantly reduces cost without sacrificing output quality.

Principles

• Draft attention's full context read becomes a bottleneck at 1M tokens.
• Local predictions for drafts do not require full million-token history.
• Target verification ensures losslessness despite draft windowing.

Method

Implement a sliding window and attention sink on the draft's KV-index construction, reducing its working set. This is achieved by limiting the block table to sink and recent tokens.

In practice

• Apply a 4K window with 64 sink tokens to Multi-Token-Prediction (MTP) draft heads.
• Reclaim unused draft KV memory using a compact ring buffer.
• Consider adaptive draft depth policies for optimal performance.

Topics

• Speculative Decoding
• LLM Inference Optimization
• KV Cache Management
• Attention Mechanisms
• Long Context Windows
• Memory Efficiency

Code references

• avalliappan-nvidia/windowed-mtp-b200
• deepseek-ai/DeepSpec
• sgl-project/sglang

Best for: MLOps Engineer, AI Engineer, NLP Engineer, AI Scientist, Machine Learning Engineer, AI Architect

Related on AIssential

• Windowed-MTP: Removing the Full-Context Draft-KV Tax at Million-Token Context — Speculative decoding's KV tax at million-token contexts can be eliminated by windowing only the draft attention.
• Speculoos…No, Speculative Decoding: The Trick That Made My Old MacBook 3x Faster — Speculative decoding accelerates LLM inference by using a small draft model to predict tokens, verified by a larger model in parallel.
• PARTREP: Learning What to Repeat for Decoder-only LLMs — PartRep selectively repeats informative tokens to improve LLM reasoning efficiency by mitigating asymmetric information flow.
• Shipping LLMs (Part 2/6): What’s Actually in Your KV Cache? — The KV cache stores per-token key/value tensors, significantly impacting LLM memory consumption and context window cost.
• Lynx: Progressive Speculative Quantization for accelerating KV Transfer in Long-Context Inference — Lynx accelerates LLM long-context inference by progressively transferring KV cache bits, enabling speculative decoding with high accuracy.
• LLM Inference and Optimization: Fundamentals, Bottlenecks, and Techniques — Optimizing LLM inference requires understanding distinct prefill and decode phases, leveraging KV caching, and applying advanced techniques.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.