Trees from Marginals: Autoregressive drafting with factorized priors

2026-07-07 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

Weaver is a lightweight autoregressive adapter designed to enhance speculative decoding for autoregressive language models. It addresses the limitation of factorized draft models, which, despite their efficiency in predicting future-token marginals in parallel, suffer from sharply degrading acceptance rates due to their independence assumption as the speculative budget increases. Weaver constructs proposal trees from the top-K marginals of a factorized drafter, thereby restoring conditional dependencies between proposed tokens without requiring a full-vocabulary projection. The system also introduces a rollback-free tree-verification algorithm and optimized CUDA kernels implemented in SGLang to support fast verification for models with Gated Delta Net layers. This combined approach achieves a 4.37-fold speedup over standard autoregressive decoding and outperforms the DFlash baseline by 24.7%.

Key takeaway

For Machine Learning Engineers optimizing autoregressive language model inference, adopting Weaver could significantly boost your system's throughput. Its ability to restore conditional dependencies in factorized draft models, combined with a rollback-free tree-verification algorithm and SGLang CUDA kernels, offers a substantial 4.37-fold speedup over traditional autoregressive decoding. Consider integrating Weaver to improve the interactivity and efficiency of your large language model applications, especially when using Gated Delta Net layers.

Key insights

Weaver improves speculative decoding by restoring conditional dependencies in factorized draft models, boosting acceptance rates.

Principles

• Factorized draft models are efficient but their independence assumption degrades speculative decoding acceptance rates.
• Restoring conditional dependencies between proposed tokens is crucial for effective speculative decoding.

Method

Weaver constructs proposal trees from top-K marginals of a factorized drafter, then uses a rollback-free tree-verification algorithm with optimized CUDA kernels for fast verification.

In practice

• Implement optimized CUDA kernels in SGLang for fast tree verification.
• Utilize Weaver to restore conditional dependencies in factorized draft models.

Topics

• Speculative Decoding
• Autoregressive Models
• Language Models
• Factorized Draft Models
• Weaver
• SGLang
• CUDA Kernels

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Domino: Decoupling Causal Modeling from Autoregressive Drafting in Speculative Decoding — Domino accelerates LLM inference by decoupling causal modeling from autoregressive drafting, using parallel generation refined by causal information.
• TAPS: Target-Aware Prefix Tree Selection for Diffusion-Drafted Speculative Decoding — TAPS optimizes speculative decoding by using target-aware prefix selection to improve acceptance-cost tradeoffs in diffusion-drafted trees.
• Speculative Refinement: A Hybrid Autoregressive Diffusion Decoding Strategy and Its Behavior Across Benchmarks — Speculative Refinement is a training-free hybrid AR-diffusion decoding method, revealing critical evaluation challenges for multi-stage generation systems.
• Why MoE models get more from speculative decoding - Cohere — MoE sparsity and expert routing significantly enhance speculative decoding performance, especially at moderate batch sizes.
• JetFlow: Breaking the Scaling Ceiling of Speculative Decoding with Parallel Tree Drafting — JetFlow breaks speculative decoding's scaling ceiling by combining one-forward drafting efficiency with branch-wise causal conditioning.
• Sampling from Flow Language Models via Marginal-Conditioned Bridges — FLMs' token-wise posterior marginals enable a more principled, training-free sampling method for improved text generation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.