Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation

2026-06-08 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Expert, short

Summary

A new method, On-Policy Diffusion Language Model (OPDLM), facilitates the data-efficient transformation of autoregressive language models (ARLMs) into diffusion language models (DLMs). Traditional approaches for this conversion suffer from two key distribution shifts: one from changing the training objective and another from a train-inference mismatch inherent in standard DLMs. OPDLM addresses these by employing On-Policy Distillation (OPD). In this self-OPD process, a student ARLM, modified with bidirectional attention, generates its own trajectories, while the original, frozen ARLM acts as a teacher, distilling knowledge by providing target logits on these generated sequences. This on-policy training eliminates the train-inference mismatch and significantly improves knowledge retention from the initial ARLM. Empirical evaluations demonstrate that OPDLM achieves strong performance across various tasks, requiring 15x to 7,000x fewer training tokens compared to pretraining DLMs from scratch, effectively making DLM transformation a form of ARLM post-training.

Key takeaway

For machine learning engineers developing diffusion language models or converting existing autoregressive models, OPDLM offers a highly data-efficient pathway. You can significantly reduce training token requirements by 15x to 7,000x, avoiding the prohibitive costs of DLM pretraining. Consider integrating On-Policy Distillation to mitigate distribution shifts and train-inference mismatch, ensuring more robust and performant DLMs from your existing ARLM assets. This approach positions DLM transformation as a practical post-training step.

Key insights

On-Policy Distillation efficiently transforms ARLMs into DLMs by resolving distribution shifts and train-inference mismatch.

Principles

• Address distribution shifts in model transformations.
• On-policy training improves inference alignment.
• Distillation retains knowledge from source models.

Method

OPDLM trains a bidirectional ARLM student using self-OPD, where a frozen ARLM teacher provides target logits on student-generated trajectories.

In practice

• Convert ARLMs to DLMs with minimal data.
• Reduce DLM pretraining computational costs.
• Improve DLM inference consistency.

Topics

• Autoregressive Language Models
• Diffusion Language Models
• On-Policy Distillation
• Model Transformation
• Data Efficiency
• Knowledge Distillation

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation — On-Policy Distillation efficiently transforms ARLMs into DLMs by addressing distribution shifts and retaining knowledge.
• Distribution Corrected Offline Data Distillation for Large Language Models — Offline distillation can be improved by correcting teacher-student distribution drift without costly online sampling.
• MemDLM: Memory-Enhanced DLM Training — MemDLM uses bi-level optimization and parametric memory to align DLM training with multi-step inference, improving convergence and long-context understanding.
• OPRD: On-Policy Representation Distillation — OPRD lifts on-policy distillation to hidden-state space, providing deterministic, richer supervision and eliminating output-space limitations.
• On-policy distillation: one of the hottest terms on PapersWithCode [R] — On-policy distillation corrects specific model errors by using a hint model to guide probability adjustments without new rollouts.
• AI 101: "On-Policy Distillation Zeitgeist" — Self-distillation offers a scalable, on-policy method for LLMs to refine reasoning by self-critique and dense feedback.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.