Windowed-MTP: Removing the Full-Context Draft-KV Tax at Million-Token Context

2026-07-23 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, quick

Summary

Windowed-MTP is a novel, training-free technique designed to eliminate the performance bottleneck of Multi-Token-Prediction (MTP) draft heads in speculative decoding at million-token contexts. Traditional MTP drafts incur a significant "KV tax" by performing full attention over the entire KV cache, leading to linearly increasing costs that can negate speculative decoding benefits, especially with deep drafts or hybrid attention targets. Windowed-MTP addresses this by applying a StreamingLLM-style sliding window and attention sink exclusively to the draft's attention mechanism, while preserving full-attention verification. This approach bounds the draft's KV working set to a constant, reducing KV entries by approximately 99% at 1M context. Benchmarks on Qwen GDN-MoE 35B/122B and Mamba2-hybrid NoPE 120B models show a +28% to +44% reduction in per-decode-step cost, improving end-to-end latency without compromising output quality.

Key takeaway

For machine learning engineers optimizing large language model inference at million-token contexts, implementing Windowed-MTP offers a significant performance uplift. You can achieve a +28% to +44% reduction in per-decode-step cost and improve end-to-end latency without retraining or sacrificing output quality. Consider integrating this training-free, drop-in solution to manage KV cache growth and reclaim memory, especially when deploying models like Qwen GDN-MoE or Mamba2-hybrid architectures.

Key insights

Speculative decoding's KV tax at million-token contexts can be eliminated by windowing only the draft attention.

Principles

• Draft attention can be windowed without affecting target verification.
• Constant KV working sets improve large context inference efficiency.
• Reclaiming unread KV reduces memory footprint.

Method

Apply a StreamingLLM-style sliding window and attention sink to the speculative decoding draft's attention mechanism, leaving the full-attention target verification intact.

In practice

• Integrate Windowed-MTP as a drop-in, training-free optimization.
• Reduce GPU memory usage by reclaiming unread draft KV.
• Improve LLM inference latency at 1M+ token contexts.

Topics

• Speculative Decoding
• Multi-Token Prediction
• KV Cache Optimization
• Large Language Models
• Context Windowing
• LLM Inference Performance

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Windowed-MTP: Removing the Full-Context Draft-KV Tax at Million-Token Context — Windowing the speculative decoding draft's attention at long contexts significantly reduces cost without sacrificing output quality.
• KV Cache Demystified: Speeding Up Large Language Models — KV cache optimizes LLM inference by storing and reusing attention keys and values, preventing redundant computations.
• ADaPT: Token-Level Decoupling for Efficient Large Reasoning Models — ADaPT decouples efficiency and correctness in large reasoning models via a token-level dual-process framework.
• Lynx: Progressive Speculative Quantization for accelerating KV Transfer in Long-Context Inference — Lynx accelerates LLM long-context inference by progressively transferring KV cache bits, enabling speculative decoding with high accuracy.
• Unlocking Parallelism in Autoregressive Language Models via Speculative Decoding with Progressive Tree Drafting — Progressive Tree Drafting (PTD) accelerates LLM inference by enabling structured, parallel exploration of semantic paths.
• Moment-KV: Momentum-Based Decode-Time KV Cache Compression for Long Generation — Moment-KV improves LLM long-generation fidelity by using momentum-driven attention aggregation for decoding-time KV cache compression.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.