Supercharging LLM inference on Google TPUs: Achieving 3X speedups with diffusion-style speculative decoding

2026-05-04 · Source: Google Developers Blog - AI · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Cloud Computing & IT Infrastructure · Depth: Expert, medium

Summary

Researchers at UCSD, led by Hao Zhang, have successfully implemented DFlash, a diffusion-style speculative decoding method, on Google TPUs, achieving significant speedups for Large Language Model (LLM) inference. This novel approach moves beyond traditional autoregressive speculative decoding's sequential token prediction, instead using a block diffusion mechanism to generate an entire block of candidate tokens in a single forward pass (O(1) complexity). Integrated into the open-source vLLM TPU inference ecosystem, DFlash demonstrated an average 3.13x increase in tokens per second on TPU v5p, with peak speedups of nearly 6x for complex math tasks. In a head-to-head comparison, DFlash achieved a 2.29x end-to-end serving speedup against EAGLE-3's 1.30x gain on TPU v5p using Llama-3.1-8B. The implementation required overcoming technical hurdles like a "dual-cache" solution for attention, intelligent context management with power-of-2 padding, and bridging metadata gaps for stateful diffusion logic.

Key takeaway

For AI Engineers optimizing LLM inference on Google TPUs, adopting DFlash's diffusion-style speculative decoding is critical. Your teams can achieve over 3x average speedup and nearly 6x for math tasks by integrating this open-source solution into vLLM. Focus on improving draft quality rather than merely increasing block size, as verification costs are constant on high-end hardware, and consider the predictability of your specific tasks to maximize performance gains.

Key insights

Block diffusion speculative decoding offers substantial LLM inference speedups by generating token blocks in a single pass.

Principles

• Verification cost is constant for wide blocks on high-end accelerators.
• Improving draft quality is 2-3x more valuable than increasing block size.
• Task predictability directly influences speculative decoding speedup.

Method

DFlash integrates a block diffusion mechanism into vLLM TPU inference, using a dual-cache for attention, power-of-2 padding for context, and strict metadata synchronization to enable parallel block generation and verification.

In practice

• Use DFlash for 3.13x average speedup on TPU v5p.
• Prioritize draft quality over block size for performance gains.
• Apply DFlash for math and coding tasks for highest speedups.

Topics

• DFlash
• Speculative Decoding
• Google TPUs
• LLM Inference Acceleration
• Block Diffusion

Code references

• vllm-project/tpu-inference
• google/tunix
• AI-Hypercomputer/maxtext

Best for: AI Engineer, NLP Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, AI Hardware Engineer

Related on AIssential

• DFlash: Block Diffusion for Flash Speculative Decoding — DFlash uses a block diffusion model for parallel drafting to significantly accelerate LLM inference.
• Speculative Decoding: How LLMs Generate Text 3x Faster — Speculative Decoding accelerates LLM inference by using a smaller model to draft tokens and a larger model to verify them in parallel.
• S2D2: Fast Decoding for Diffusion LLMs via Training-Free Self-Speculation — S2D2 accelerates block-diffusion LLMs by reusing the same model as both drafter and autoregressive verifier.
• Train and Run DFlash Speculative Decoding — DFlash accelerates LLM inference by predicting and verifying entire blocks of future tokens in a single pass.
• Structuring The Future: Diffusion LLM Speculative Decoding via Calibrated Draft Graphs — Spiffy accelerates dLLM inference up to 7.9x by auto-speculative decoding with calibrated directed draft graphs, preserving output quality.
• 8x Faster Inference: DFlash with Block Diffusion Draft Trees — DDTree and IceCache offer distinct approaches to enhance LLM inference efficiency through improved speculative decoding and KV-cache management.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Google Developers Blog - AI.