Where Should Diffusion Enter a Language Model? Geometry-Guided Hidden-State Replacement

2026-05-15 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, extended

Summary

DiHAL (Diffusion-Transformer Hybrid Architecture for Language Generation) is a novel approach that integrates continuous diffusion models into pretrained autoregressive Transformers to improve language generation. It addresses the challenge of applying diffusion to discrete text by focusing on reconstructing continuous hidden states within the Transformer, rather than directly generating tokens. DiHAL employs a "Locate-and-Replace" strategy: it first identifies "diffusion-friendly" internal layers using geometry-based proxies like local compactness, global stiffness, and effective rank. Then, it replaces the lower Transformer layers with a conditional diffusion bridge that reconstructs the selected layer's hidden state, while retaining the upper layers and original LM head for token prediction. Experiments on 8B-scale backbones, including Llama-3.1-8B-Instruct and Qwen3-8B, demonstrate that the geometry score effectively predicts bridgeable layers, typically embedding-adjacent ones, and that hidden-state recovery improves generative perplexity and diversity compared to continuous diffusion baselines under matched training budgets.

Key takeaway

For research scientists exploring hybrid generative models, DiHAL suggests that focusing continuous diffusion on internal Transformer hidden states, guided by geometric properties, can yield more effective language generation than direct token recovery. You should consider analyzing the geometric properties of intermediate representations in your models to identify optimal integration points for diffusion components, potentially improving perplexity and diversity without needing to train a full standalone diffusion LM.

Key insights

Geometry-guided diffusion within Transformer hidden states improves continuous language generation by avoiding direct token recovery.

Principles

• Diffusion-friendly spaces are easy to denoise and stable.
• Curvature and low effective dimensionality are key geometric properties.
• Internal hidden states offer better diffusion targets than token embeddings.

Method

DiHAL uses geometry-based proxies (local curvature, global monotonicity, effective rank) to score and select an optimal Transformer layer, then replaces lower layers with a conditional diffusion bridge to reconstruct that layer's hidden state.

In practice

• Use geometry scores to identify optimal diffusion insertion points.
• Focus diffusion on internal hidden states, not just token embeddings.
• Repurpose UNet architectures for hidden-state denoising.

Topics

• DiHAL
• Diffusion-Transformer Hybrid
• Hidden-State Geometry
• Language Model Denoising
• Layer-Wise Proxy

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Where Should Diffusion Enter a Language Model? Geometry-Guided Hidden-State Replacement — Geometry-guided hidden-state replacement improves diffusion language models by optimizing insertion points within transformers.
• Beyond Masks: Efficient, Flexible Diffusion Language Models via Deletion-Insertion Processes — DID improves diffusion language models by replacing masking with deletion-insertion for efficiency and flexibility.
• Language Diffusion Models are Associative Memories Capable of Retrieving Unseen Data — UDDMs behave as associative memories, exhibiting a dataset-size-dependent memorization-to-generalization transition detectable via conditional entropy.
• Nemotron-TwoTower: Diffusion Language Modeling with Pretrained Autoregressive Context — Decoupling context and denoising in diffusion models significantly boosts generation throughput while maintaining quality.
• On the Role of Discreteness in Diffusion LLMs — Applying continuous diffusion to discrete text requires addressing fundamental structural trade-offs.
• Teaching Diffusion to Speculate Left-to-Right — Aligning bidirectional diffusion drafters with left-to-right autoregressive verification significantly boosts speculative decoding efficiency.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.