On-Policy Delta Distillation

2026-07-16 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

On-Policy Delta Distillation (OPD^2) is a novel post-training method in reinforcement learning designed to enhance reasoning capabilities in large language models. This approach introduces a "delta signal" as a distillation reward, which fundamentally differs from directly imitating a teacher model's output distribution. The delta signal is precisely defined as the difference between a teacher model and its base model prior to instruction tuning for reasoning. This mechanism effectively captures the specific changes induced by reasoning tuning, providing a more direct and potent signal for transferring these capabilities. Empirical evidence demonstrates that OPD^2 substantially improves on-policy distillation, consistently outperforming conventional methods across mathematics, science, and code-reasoning benchmarks, achieving strong performance with only a short post-training period. Code for OPD^2 will be made available.

Key takeaway

For AI Scientists and Machine Learning Engineers focused on improving LLM reasoning, OPD^2 offers a superior post-training method. By utilizing a "delta signal" that directly targets reasoning-induced changes, you can achieve strong performance on mathematics, science, and code-reasoning tasks with significantly shorter post-training periods. Consider integrating OPD^2 into your model development pipeline to enhance efficiency and effectiveness in transferring complex reasoning capabilities.

Key insights

On-Policy Delta Distillation (OPD^2) leverages a "delta signal" to directly transfer reasoning capabilities by focusing on changes induced by instruction tuning.

Principles

• Delta signal captures reasoning tuning changes.
• Distill difference, not direct output imitation.

Method

OPD^2 defines a delta signal as the difference between a teacher model and its pre-instruction-tuned base model, using this as a token-level distillation reward for transferring reasoning.

In practice

• Implement delta signal for RL distillation.
• Evaluate OPD^2 on reasoning benchmarks.

Topics

• On-Policy Delta Distillation
• Reinforcement Learning
• Large Language Models
• Model Distillation
• Reasoning Capabilities
• Post-training

Code references

• naver-ai/opd2

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Demystifying On-Policy Distillation: Roles, Pathologies, and Regulations — OPD acts as an exploration catalyst, effective only with high-quality, well-regulated guiding signals.
• H$^2$SD: Hybrid Hindsight Self-Distillation — H$^2$SD improves LLM reasoning by adaptively applying self-distillation based on trajectory correctness.
• Bridging Reasoning Trajectories in On-Policy Distillation via Near-Future Guidance — On-Policy Distillation's token-level learning struggles with reasoning divergence; near-future trajectory guidance improves alignment and performance.
• On-Policy Context Distillation for Language Models — OPCD enables language models to internalize in-context knowledge by training on self-generated trajectories against a context-conditioned teacher.
• Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe — OPD success hinges on compatible thinking patterns and genuinely new teacher knowledge, not just higher scores.
• Breaking the Tokenizer Barrier: On-Policy Distillation across Model Families — Cross-tokenizer On-Policy Distillation (OPD) enables knowledge transfer between LLMs with different tokenizers via a precise token-mapping algorithm.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.