WhiFlash: Accelerating Speculative Decoding with Token-Level Cross-Paradigm Routing

2026-06-05 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

WhiFlash is a novel speculative decoding (SD) method designed to accelerate large language model (LLM) inference, particularly for complex agentic workloads. It addresses the limitations of current SD approaches that rely on static drafting paradigms, which lead to fluctuating accuracy. WhiFlash introduces the first cross-paradigm SD, unifying autoregressive and diffusion-based parallel drafting under a token-level controller. This fine-grained routing uses either an entropy-based or learned neural policy, balancing token gain and latency. To enable high-frequency switching, WhiFlash incorporates cache-management optimizations like Lazy Catch-up and KV-only Prefill, reducing switching overhead to below 7% of per-round latency. This approach yields throughput gains of up to 69.6% over the autoregressive EAGLE-3 and 37.3% over the diffusion-based DFlash.

Key takeaway

For AI Scientists and Machine Learning Engineers optimizing LLM inference for agentic workloads, WhiFlash presents a significant advancement. You should evaluate integrating cross-paradigm speculative decoding to overcome the limitations of static drafting methods. This approach, with its token-level routing and cache optimizations, can deliver substantial throughput gains, up to 69.6% over existing autoregressive solutions, by dynamically adapting to output characteristics. Consider its novel cache management for high-frequency model switching.

Key insights

WhiFlash unifies autoregressive and diffusion-based speculative decoding via token-level routing to overcome static paradigm limitations.

Principles

• Drafting accuracy fluctuates significantly within a sequence.
• Cross-paradigm routing improves speculative decoding.
• Fine-grained control balances token gain and latency.

Method

WhiFlash employs a token-level controller with entropy-based or learned neural policies for fine-grained routing, supported by Lazy Catch-up and KV-only Prefill cache optimizations.

In practice

• Implement token-level routing for dynamic SD.
• Optimize cache management for frequent model switching.
• Combine distinct drafting architectures for higher acceptance.

Topics

• Speculative Decoding
• Large Language Models
• LLM Inference
• Cross-Paradigm Routing
• Cache Management
• Throughput Optimization

Best for: NLP Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• DFlash: Block Diffusion for Flash Speculative Decoding — DFlash uses a block diffusion model for parallel drafting to significantly accelerate LLM inference.
• FLy: A New Paradigm for Speculative Decoding — Accepting Semantically Correct Drafts Beyond Exact Match — FLy accelerates LLM inference by accepting semantically valid drafts, overcoming exact-match limitations without training.
• Speculoos…No, Speculative Decoding: The Trick That Made My Old MacBook 3x Faster — Speculative decoding accelerates LLM inference by using a small draft model to predict tokens, verified by a larger model in parallel.
• VIA-SD: Verification via Intra-Model Routing for Speculative Decoding — Multi-tier speculative decoding with a slim-verifier significantly improves LLM inference efficiency by reducing full-model recomputations.
• Train and Run DFlash Speculative Decoding — DFlash accelerates LLM inference by predicting and verifying entire blocks of future tokens in a single pass.
• Why MoE models get more from speculative decoding - Cohere — MoE sparsity and expert routing significantly enhance speculative decoding performance, especially at moderate batch sizes.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.