Gumbel Distillation for Parallel Text Generation

2026-07-25 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, long

Summary

Gumbel Distillation is a novel technique addressing the quality degradation in parallel text generation models, which are faster than autoregressive (AR) models but struggle with complex joint token distributions. This model-agnostic method uses the Gumbel-Max trick to create a deterministic mapping from latent Gumbel noise to the output tokens of a high-performing AR teacher. Experiments on LM1B and OpenWebText datasets show Gumbel Distillation significantly improves generation quality, achieving a 30.0% improvement in MAUVE score and 10.5% reduction in generative perplexity over MDLM. It integrates seamlessly with architectures like MDLM and BD3-LM, and enhances Multi-Token Prediction models such as Medusa, improving acceptance rates by up to +37.6% on Vicuna-7B.

Key takeaway

For Machine Learning Engineers developing high-throughput text generation systems, Gumbel Distillation offers a critical solution to the quality-speed trade-off. By integrating this method, you can significantly enhance the generation quality of parallel decoders like MDLM or Medusa, achieving AR-like coherence and reasoning capabilities at accelerated inference speeds. Consider applying Gumbel Distillation to improve both fluency and factual accuracy in your non-autoregressive models.

Key insights

Gumbel Distillation improves parallel decoders by enabling them to learn complex joint token distributions from AR teachers via Gumbel noise.

Principles

• Parallel decoders struggle with joint probability distributions due to conditional independence assumptions.
• The Gumbel-Max trick enables deterministic mapping from Gumbel noise to AR teacher outputs.

Method

Gumbel Distillation uses the Gumbel-Max trick to map latent Gumbel noise to AR teacher output sequences, then trains a non-AR student to reconstruct text conditioned on this noise.

In practice

• Augment MDLM/BD3-LM input with processed Gumbel noise embeddings instead of [MASK] tokens.
• Provide processed Gumbel noise as a conditioning vector to Medusa's multi-token prediction heads.

Topics

• Gumbel Distillation
• Parallel Text Generation
• Knowledge Distillation
• Autoregressive Models
• Masked Diffusion Language Models
• Multi-Token Prediction
• Language Model Quality

Code references

• hxixixh/gumbel-distill

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• IB-Flow: Information Bottleneck-Guided CFG Distillation for Few-Step Text-to-Image Generation — IB-Flow uses Information Bottleneck-guided CFG distillation with adaptive mechanisms to achieve SOTA few-step text-to-image generation without over-conditioning.
• Multi-Mask Diffusion Language Models for Few-Step Generation — MultiMDM improves few-step language generation by preserving masking structure and enabling a drafting capability.
• SwiftAudio: Data-Efficient Caption-Only Distillation for One-Step Text-to-Audio Diffusion-based Generation — SwiftAudio enables one-step text-to-audio generation via audio-free distillation using only text captions, significantly reducing latency and data requirements.
• One-step Language Modeling via Continuous Denoising — Continuous denoising via flow-based models can surpass discrete diffusion in language model quality and speed.
• T3D: Few-Step Diffusion Language Models via Trajectory Self-Distillation with Direct Discriminative Optimization — Trajectory self-distillation with Direct Discriminative Optimization improves few-step diffusion language model efficiency.
• Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalization — BPM enables effective cross-tokenizer on-policy distillation by re-expressing teacher distributions in a shared byte space, preserving probability mass.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.