Learning Unmasking Policies for Diffusion Language Models

2026-07-02 · Source: Apple Machine Learning Research · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

Diffusion Language Models (dLLMs) are gaining parity with autoregressive models in downstream tasks and promise greater inference efficiency. A critical aspect of dLLM inference is the unmasking policy, which dictates token selection at each diffusion step. While heuristic strategies like confidence thresholding improve sample quality and token throughput over random unmasking, they require manual tuning and degrade with larger block sizes. This work proposes training unmasking policies using reinforcement learning. The masked diffusion sampling is formalized as a Markov decision process, with a lightweight policy based on a single-layer transformer mapping dLLM token confidences to unmasking decisions. Experiments show these trained policies match the performance of leading heuristics when combined with semi-autoregressive (block) generation, while outperforming them in the full-diffusion setting.

Key takeaway

For NLP Engineers optimizing Diffusion Language Model inference, consider integrating reinforcement learning for unmasking policies. Your current heuristic strategies, while effective, may face performance degradation with larger block sizes and require manual tuning. By formalizing sampling as a Markov decision process and training a lightweight transformer policy, you can achieve superior performance, especially in full-diffusion settings, enhancing both sample quality and token throughput.

Key insights

Reinforcement learning can optimize dLLM unmasking policies, outperforming heuristics and improving inference efficiency.

Principles

• Heuristic unmasking policies have tuning and scaling limitations.
• Diffusion sampling can be modeled as a Markov decision process.
• RL-trained policies can surpass fixed heuristics in dLLMs.

Method

Formalize masked diffusion sampling as a Markov Decision Process. Train a single-layer transformer policy using reinforcement learning to map dLLM token confidences to unmasking decisions.

In practice

• Apply RL to optimize dLLM inference sampling strategies.
• Consider transformer-based policies for unmasking decisions.
• Evaluate RL policies in full-diffusion and semi-autoregressive modes.

Topics

• Diffusion Language Models
• Reinforcement Learning
• Unmasking Policies
• Inference Optimization
• Markov Decision Process

Best for: Research Scientist, AI Scientist, NLP Engineer

Related on AIssential

• Reinforcement Learning from Denoising Feedback — RLDF enhances dLLM reasoning by efficiently estimating policy loss via weighted sampling of denoising feedback and clean state prediction.
• Policy Gradients: REINFORCE and Actor-Critic — Policy gradient methods directly optimize a parameterized policy to maximize expected return, suitable for continuous action spaces.
• DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Models — DiffusionOPD unifies multi-task diffusion model training via online policy distillation, improving efficiency and performance.
• SPG: Sandwiched Policy Gradient for Masked Diffusion Language Models — SPG uses both upper and lower bounds of intractable log-likelihoods to reduce bias in dLLM reinforcement learning.
• Trust-Region Diffusion Policies for Massively Parallel On-Policy RL — TruDi enables stable, effective on-policy training of expressive diffusion policies in massively parallel RL using a KL-divergence trust-region.
• Tool-Aware Optimization with Entropy Guidance for Efficient Agentic Reinforcement Learning — TAO-RL stabilizes agentic reinforcement learning by filtering tool-use trajectories and guiding exploration with entropy for efficient LLM policy optimization.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Apple Machine Learning Research.