Discrete Diffusion with Sample-Efficient Estimators for Conditionals

2026-07-22 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences · Depth: Expert, extended

Summary

The paper introduces a discrete denoising diffusion framework that employs a sample-efficient Neural Interaction Screening Estimator (NeurISE) for generative modeling over discrete state spaces. This approach estimates single-site conditional probabilities to parameterize the reverse diffusion process, rather than approximating a discrete score function. Using a round-robin noising scheme, the method was tested on synthetic Ising models (25 variables), binarized MNIST, D-Wave quantum annealer datasets (2000 qubits), multi-alphabet Potts models (L=2, L=3), and quantum tomography data of GHZ states (20 qubits). Experiments show NeurISE diffusion outperforms existing ratio-based and ELBO-based methods like D3PM and SEDD in total variation, cross-correlation, and kernel density estimation metrics, especially with varying training sample sizes up to 10^5.

Key takeaway

For Machine Learning Engineers developing generative models for discrete data, this framework offers a robust alternative to score-based methods. You should consider implementing discrete diffusion with NeurISE and a round-robin noising scheme, as it demonstrates superior performance in capturing complex dependencies and distributional accuracy on various datasets, including quantum and image data. This approach simplifies the reverse process by focusing on local conditionals, potentially leading to more stable and sample-efficient training.

Key insights

Discrete diffusion can be effectively modeled by estimating single-site conditional probabilities, not discrete score functions.

Principles

• Reverse diffusion kernels depend on ratios of single-site conditional distributions.
• Total variation error in sampling decomposes into mixing error and kernel estimation error.
• Round-robin noising reduces the number of ratios to learn per time step.

Method

The method uses a round-robin noising scheme and the Neural Interaction Screening Estimator (NeurISE) to learn single-site conditional probabilities, which then parameterize the reverse diffusion process for generative sampling.

In practice

• Apply NeurISE to estimate local conditionals for discrete diffusion models.
• Consider round-robin noising for sample-efficient discrete diffusion.
• Evaluate discrete diffusion models using total variation and cross-correlation metrics.

Topics

• Discrete Diffusion Models
• Generative Modeling
• Neural Interaction Screening Estimator
• Round-Robin Noising
• Ising Models
• Quantum Annealing Data
• MNIST Dataset

Code references

• cloneofsimo/d3pm

Best for: AI Scientist, Machine Learning Engineer, Research Scientist

Related on AIssential

• Forward-Learned Discrete Diffusion: Learning how to noise to denoise faster — Learning the forward noising process in discrete diffusion enables few-step, high-quality generation.
• Mean-to-Score Discrete Diffusion: Posterior-Mean Denoisers for Score Entropy — Mean-to-Score (M2S) improves discrete diffusion by ensuring Bayes realizability through posterior mean prediction, outperforming prior methods.
• Low-dimensional adaptation of diffusion models: Convergence in total variation — Diffusion models adapt to low-dimensional data, accelerating sampling with provable k/ε convergence in total variation.
• Discrete Diffusion Models: A Unified Framework from Tokenization to Generation — The framework unifies discrete diffusion models by focusing on their underlying discrete state space construction.
• Provably Learning Diffusion Models under the Manifold Hypothesis: Collapse and Refine — The score function's singularity at low noise drives a two-stage "collapse-and-refine" mechanism for efficient manifold learning in diffusion models.
• Temporal Difference Learning for Diffusion Models — Temporal difference learning improves diffusion model consistency and sample quality, especially for few-step sampling.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.