A Survey of On-Policy Distillation for Large Language Models

2026-06-19 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, extended

Summary

A comprehensive survey introduces On-Policy Distillation (OPD) for Large Language Models (LLMs), addressing the exposure bias inherent in traditional off-policy methods. Off-policy distillation trains student models on static, teacher-generated data, leading to compounding prediction errors during autoregressive inference. OPD, grounded in interactive imitation learning, enables students to generate their own sequences and receive iterative teacher feedback. This survey unifies the fragmented OPD literature through an $f$-divergence framework over on-policy samples. It categorizes methods across three orthogonal dimensions: feedback signal (logit-based, outcome-based, self-play), teacher access (white-box, black-box, teacher-free), and loss granularity (token-level, sequence-level, hybrid). The analysis covers representative techniques like GKD, MiniLLM, and SPIN, examines industrial applications such as DeepSeek-R1's transfer of reasoning from a 671-billion-parameter teacher to 1.5-70 billion parameter students, and identifies future research directions.

Key takeaway

For machine learning engineers deploying smaller LLMs, recognize that traditional off-policy distillation introduces exposure bias, limiting performance on multi-step generation. You should explore On-Policy Distillation (OPD) to mitigate this by allowing your student models to learn from their own generated outputs. Consider implementing hybrid granularity losses and adaptively choosing $f$-divergences like Reverse KL for reasoning tasks to achieve more robust and accurate capability transfer.

Key insights

OPD overcomes off-policy distillation's exposure bias by enabling LLMs to learn from self-generated trajectories with teacher feedback.

Principles

• Off-policy training creates train-test mismatch.
• On-policy feedback reduces autoregressive error.
• Divergence choice shapes student mode-seeking.

Method

OPD involves student LLMs generating trajectories, then receiving teacher feedback (logit, outcome, or self-play) on these outputs, iteratively refining the policy.

In practice

• Match $f$-divergence to task (e.g., Reverse KL for math).
• Combine token and sequence losses for complex reasoning.
• Leverage privileged information in self-distillation.

Topics

• On-Policy Distillation
• Large Language Models
• Knowledge Distillation
• Exposure Bias
• f-Divergence
• Self-Distillation

Best for: Research Scientist, AI Engineer, NLP Engineer, AI Scientist, Machine Learning Engineer, AI Architect

Related on AIssential

• On-policy distillation: one of the hottest terms on PapersWithCode [R] — On-policy distillation corrects specific model errors by using a hint model to guide probability adjustments without new rollouts.
• AI 101: "On-Policy Distillation Zeitgeist" — Self-distillation offers a scalable, on-policy method for LLMs to refine reasoning by self-critique and dense feedback.
• OPRD: On-Policy Representation Distillation — OPRD lifts on-policy distillation to hidden-state space, providing deterministic, richer supervision and eliminating output-space limitations.
• When Context Returns: Toward Robust Internalization in On-Policy Distillation — Reintroducing privileged context to on-policy distilled models can degrade performance, a problem addressed by a new consistency regularizer ensuring context removability.
• Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation — On-Policy Distillation efficiently transforms ARLMs into DLMs by addressing distribution shifts and retaining knowledge.
• Distribution Corrected Offline Data Distillation for Large Language Models — Offline distillation can be improved by correcting teacher-student distribution drift without costly online sampling.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.