Likelihood Matching for Diffusion Models

2026-07-14 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Data Science & Analytics · Depth: Expert, long

Summary

Likelihood Matching is a novel approach for training diffusion models, directly maximizing data likelihood by establishing an equivalence with the likelihood along the reverse diffusion sample path. Unlike traditional score matching, this method approximates reverse transition densities using a quasi-likelihood, which matches both conditional mean and covariance with Gaussian distributions. This requires estimating both the score and Hessian functions for diffusion generation. A new stochastic sampler is introduced, leveraging both estimated score and Hessian information. The method provides non-asymptotic convergence guarantees, with reverse step error scaling at O(d^3log^4.5T/T), score estimation error at O(sqrt(log T)), and Hessian estimation error at O(log T/sqrt(T)). Consistency of the quasi-maximum likelihood estimation is theoretically proven. Empirical results on synthetic datasets and image benchmarks like MNIST, CIFAR10, CelebA, and LSUN demonstrate its superior performance and faster convergence compared to Score Matching, particularly with fewer sampling steps.

Key takeaway

For machine learning engineers developing diffusion models, you should consider Likelihood Matching as a superior training objective. This method directly maximizes data likelihood by incorporating both score and covariance matching, offering improved accuracy and faster convergence compared to traditional score matching. Implement this approach, particularly when generating high-fidelity samples or performing parameter estimation with limited sampling steps, to achieve better results and efficiency.

Key insights

Likelihood Matching directly optimizes diffusion model training by integrating score and covariance matching via quasi-likelihood.

Principles

• Maximize data likelihood directly, not an upper bound.
• Reverse diffusion path likelihood equals original data likelihood.
• Incorporate both score and Hessian information for accuracy.

Method

Approximates reverse transition densities with Gaussian quasi-likelihood, matching conditional mean and covariance. Optimizes score and Hessian functions using neural networks. Employs a stochastic sampler.

In practice

• Apply to image synthesis (MNIST, CIFAR10, CelebA).
• Use for parameter estimation in mixture models.
• Outperforms Score Matching with fewer sampling steps.

Topics

• Diffusion Models
• Likelihood Matching
• Score Matching
• Quasi-Maximum Likelihood Estimation
• Generative Models
• Image Synthesis
• Stochastic Samplers

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• From Score Approximation to Distribution Approximation in Score-Based Diffusion Models — A rigorous theorem links neural network score function approximation directly to generated probability distribution approximation in diffusion models.
• Inverse Entropic Optimal Transport Solves Semi-supervised Learning via Data Likelihood Maximization — Inverse entropic optimal transport provides a unified likelihood maximization framework for semi-supervised conditional distribution learning using mixed data.
• Probabilistic ML - 21 - Diffusion Models — Diffusion models transform complex distributions into simple Gaussians via iterative noise, then reverse the process to generate data.
• Conservation Laws for Diffusion Models — Conservation laws based on GEXIT functions characterize diffusion model likelihood via local information-theoretic derivatives.
• Nonparametric Estimation of a Factorizable Density using Diffusion Models — Diffusion models can implicitly estimate factorizable densities, achieving minimax optimal rates.
• Optimizing Diffusion Priors with a Single Observation — Optimize diffusion priors from a single observation by combining them into a product-of-experts and maximizing Bayesian evidence.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.