CANDI: Hybrid Discrete-Continuous Diffusion Models

2026-07-14 · Source: stat.ML updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, extended

Summary

CANDI (Continuous ANd DIscrete diffusion) is a novel hybrid diffusion model designed to overcome the limitations of applying continuous diffusion directly to discrete data, such as text. Researchers from Purdue University and Google DeepMind identified a "temporal dissonance" caused by how Gaussian noise corrupts discrete data, specifically through "discrete identity corruption" and "continuous rank degradation," which scale differently with vocabulary size. CANDI addresses this by introducing a structured noising kernel that decouples these corruption mechanisms, allowing for simultaneous learning of discrete conditional structure and continuous geometry. This approach enables benefits like classifier-based guidance via simple gradient addition and demonstrates improved text generation, outperforming masked diffusion at low Numbers of Function Evaluations (NFE), with perplexity scores like 71.43 at NFE=16 and 50.58 at NFE=32, using temperatures around τ=.85.

Key takeaway

For NLP engineers developing generative models for discrete data, you should consider hybrid diffusion approaches like CANDI. This framework resolves the "temporal dissonance" issue in continuous diffusion, allowing your models to simultaneously learn discrete conditional structure and continuous gradients. Implementing CANDI can enable more effective classifier-based guidance and improve text generation quality, especially in compute-constrained settings requiring low Numbers of Function Evaluations.

Key insights

Continuous diffusion fails on discrete data due to "temporal dissonance" between discrete identity corruption and continuous rank degradation.

Principles

• Discrete identity corruption scales exponentially with vocabulary size.
• Continuous rank degradation is independent of vocabulary size.
• Effective discrete diffusion requires learning both conditional structure and continuous gradients.

Method

CANDI uses a structured noising kernel with discrete masking to preserve clean positions and Gaussian noise for others. This decouples corruption, ensuring both scale linearly with time for stable learning.

In practice

• Enable classifier-based guidance via simple gradient addition.
• Improve generative quality at low NFE for text generation.
• Apply to discrete sequence generation with large vocabularies.

Topics

• Diffusion Models
• Discrete Data Generation
• Text Generation
• Hybrid Models
• Score-based Generative Models
• Token Identifiability

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• CRoCoDiL: Continuous and Robust Conditioned Diffusion for Language — CRoCoDiL improves text generation by fusing continuous semantic guidance with discrete masked diffusion models.
• Discrete Diffusion Models: A Unified Framework from Tokenization to Generation — The framework unifies discrete diffusion models by focusing on their underlying discrete state space construction.
• Continuous Diffusion Scales Competitively with Discrete Diffusion for Language — Continuous diffusion language models, when likelihood-trained, can scale competitively with discrete diffusion models.
• The Sequence Knowledge #862: Learning About Text Diffusion Models — Autoregressive LLMs suffer from sequential generation pathologies, suggesting text diffusion models offer a promising alternative.
• DiffMath: Symbol- and Graph-Aware Latent Diffusion Transformer for Handwritten Mathematical Expression Generation — DiffMath generates HMEG without explicit spatial supervision by using LaTeX structure and a structured latent diffusion model.
• One-step Language Modeling via Continuous Denoising — Continuous denoising via flow-based models can surpass discrete diffusion in language model quality and speed.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by stat.ML updates on arXiv.org.