Uniform Diffusion Models Revisited: Leave-One-Out Denoiser and Absorbing State Reformulation

2026-05-21 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

This paper revisits Uniform Diffusion Models (UDM), identifying a critical mismatch where the standard plug-in bridge parameterization is not optimized by the denoising posterior but by a leave-one-out posterior. This leave-one-out approach predicts each clean token without using its own noisy observation. The authors characterize this target and derive exact conversions among the denoiser, leave-one-out posterior, and score, allowing disentanglement of parameterization and training objective. These findings enable inference improvements, including an informed predictor-corrector sampler and improved temperature sampling. Furthermore, an absorbing-state reformulation of uniform diffusion is introduced, which simplifies denoising posteriors and sampling operations. On language modeling tasks, leave-one-out parameterizations consistently enhance UDM generation, and the absorbing construction matches or surpasses masked diffusion, suggesting performance gaps are driven by design choices rather than marginals.

Key takeaway

For machine learning engineers optimizing discrete diffusion models, particularly in language modeling, you should re-evaluate your UDM parameterization and sampling strategies. Adopting leave-one-out denoisers or the proposed absorbing-state reformulation can significantly improve generation quality and inference efficiency, potentially closing the performance gap with masked diffusion models without additional training. Consider experimenting with the informed predictor-corrector sampler and improved temperature sampling for immediate gains.

Key insights

Re-evaluating Uniform Diffusion Model parameterization and sampling design is crucial for improving discrete diffusion model performance.

Principles

• Standard UDM plug-in bridge parameterization is optimized by a leave-one-out posterior, not the denoising posterior.
• The empirical performance gap between masked and uniform diffusion models stems from parameterization and sampling design.
• Exact conversions exist between the denoiser, leave-one-out posterior, and score.

Method

The paper characterizes the leave-one-out target and derives exact conversions between the denoiser, leave-one-out posterior, and score, alongside introducing an absorbing-state reformulation of uniform diffusion.

In practice

• Implement leave-one-out parameterizations to enhance UDM generation.
• Utilize an informed predictor-corrector sampler for improved inference.
• Apply improved temperature sampling based on the leave-one-out predictor.

Topics

• Discrete Diffusion Models
• Uniform Diffusion Models
• Leave-One-Out Denoiser
• Absorbing State Reformulation
• Language Modeling
• Predictor-Corrector Sampling

Code references

• samsongourevitch/rev_udm

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• One-step Language Modeling via Continuous Denoising — Continuous denoising via flow-based models can surpass discrete diffusion in language model quality and speed.
• Enhanced Low-Density Region Exploration in Classifier-Guided Diffusion Models Through Modified Reverse Diffusion Sampling — A novel sampling-time modification for classifier-guided diffusion models improves low-density region exploration without retraining.
• Diffusion Language Models: An Experimental Analysis — Diffusion Language Models' performance and efficiency are critically shaped by generation-time design choices.
• End-to-End Training for Unified Tokenization and Latent Denoising — Unified tokenization and latent diffusion can be jointly optimized in a single-stage training process.
• Uncertainty Quantification for Large Language Diffusion Models — Lightweight, zero-shot uncertainty quantification for LLDMs can achieve high performance with significantly reduced computational cost.
• [P] Micro Diffusion — Discrete text diffusion in ~150 lines of pure Python — Discrete text diffusion generates all tokens at once by iteratively unmasking from noise.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.