Mean-to-Score Discrete Diffusion: Posterior-Mean Denoisers for Score Entropy

2026-07-23 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

Mean-to-Score (M2S) is a novel parameterization for discrete diffusion models that addresses Bayes realizability issues found in methods like Score Entropy Discrete Diffusion (SEDD). SEDD's unconstrained positive score ratios often lead to score vectors incompatible with any valid clean-token posterior, causing negative pre-normalization weights during sampling. M2S predicts a clean-token posterior mean and converts it to a score via an exact kernel-dependent linear map, ensuring Bayes realizability. This construction applies to any coordinate-wise continuous-time Markov chain (CTMC) with mild support. For uniform corruption, M2S maps the probability simplex onto the bridge polytope, and for absorbing-mask corruption, it recovers MD4. In a 28.4M-parameter CIFAR-10 comparison, M2S lowered test BPD from 3.173 to 3.129 and FID-50k from 2.94 to 2.78. A 170M-parameter M2S model trained on 262B OpenWebText token slots achieved a generative PPL of 143.3 at 128 steps, outperforming pure-uniform SEDD (183.6), GIDD, and Neural CTMC baselines.

Key takeaway

For AI scientists and ML engineers developing discrete diffusion models, adopting the Mean-to-Score (M2S) parameterization can significantly enhance model stability and generative performance. M2S directly addresses Bayes realizability issues in methods like SEDD, preventing negative pre-normalization weights and yielding superior PPL and FID scores. You should investigate integrating M2S into your next-generation discrete diffusion architectures to achieve more robust and higher-quality generative results.

Key insights

Mean-to-Score (M2S) improves discrete diffusion by ensuring Bayes realizability through posterior mean prediction, outperforming prior methods.

Principles

• Bayes realizability is crucial for stable discrete diffusion.
• Projecting raw scores can mitigate negative sampling weights.
• Posterior mean prediction can ensure valid score vectors.

Method

M2S predicts a clean-token posterior mean and converts it to a score using an exact kernel-dependent linear map, applicable to coordinate-wise continuous-time Markov chains.

In practice

• Implement M2S for improved discrete diffusion performance.
• Consider score projection for existing SEDD models.
• Apply M2S to uniform or absorbing-mask corruption tasks.

Topics

• Discrete Diffusion Models
• Score Entropy Discrete Diffusion
• Mean-to-Score (M2S)
• Generative Models
• Bayes Realizability
• CIFAR-10
• OpenWebText

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• Discrete Diffusion with Sample-Efficient Estimators for Conditionals — Discrete diffusion can be effectively modeled by estimating single-site conditional probabilities, not discrete score functions.
• Where the Score Lives: A Wavelet View of Diffusion — A wavelet-based score function parameterization reveals how data moments and architecture biases influence diffusion model generation.
• DSL-LLaDA: Scaling Continuous Denoising to 8B Masked Diffusion LMs — Adapting pretrained masked diffusion LMs with continuous denoising via soft masking resolves few-step generation tradeoffs.
• The Race to Production-Grade Diffusion LLMs [Stefano Ermon] - 764 — Diffusion LLMs offer superior inference speed and cost-efficiency compared to autoregressive models for production-grade applications.
• Diffusion models recover accurate mixture weights despite score function insensitivity — Diffusion models recover accurate mixture weights by leveraging score sensitivity at intermediate noise levels, quantified by the Diffusion Score Sensitivity Index (DSSI).
• Provably Learning Diffusion Models under the Manifold Hypothesis: Collapse and Refine — The score function's singularity at low noise drives a two-stage "collapse-and-refine" mechanism for efficient manifold learning in diffusion models.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.