Denser $\neq$ Better: Limits of On-Policy Self-Distillation for Continual Post-Training

2026-07-02 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

A study revisiting on-policy self-distillation policy optimization (SDPO) for continual post-training of foundation models reveals its limitations despite prior optimistic views. Published on 2026-07-02, the research indicates that SDPO can accelerate in-domain specialization when teacher signals are stable and well-aligned. However, it struggles with out-of-distribution scenarios, exhibiting stronger forgetting and potential collapse during continual post-training. In contrast, on-policy reinforcement learning methods like GRPO adapt more conservatively and better preserve prior capabilities. Analyses show that denser self-distillation induces larger drift in both parameter and response spaces, amplifying high-frequency formatting artifacts. These findings suggest on-policy data alone is insufficient for robust continual learning.

Key takeaway

For Machine Learning Engineers implementing continual post-training for foundation models, you should reconsider on-policy self-distillation (SDPO) as a default stabilization technique. SDPO accelerates in-domain specialization but struggles with out-of-distribution scenarios and can amplify forgetting. Instead, evaluate methods like GRPO for better prior capability preservation, or ensure extremely stable teacher signals if using SDPO to mitigate drift and artifact amplification.

Key insights

On-policy self-distillation, while accelerating in-domain specialization, is insufficient for robust continual learning due to generalization issues and forgetting.

Principles

• Denser self-distillation induces larger parameter and response space drift.
• On-policy data alone is insufficient for robust continual learning.
• Stable, well-aligned teacher signals are crucial for SDPO specialization.

In practice

• Avoid dense self-distillation as a default stabilizer for continual post-training.
• Consider GRPO for more conservative adaptation and prior capability preservation.
• Ensure stable, token-level teacher supervision for SDPO specialization.

Topics

• Continual Learning
• Self-Distillation
• Foundation Models
• On-Policy Learning
• SDPO
• GRPO

Code references

• Moenupa/SDPO-CL

Best for: Research Scientist, NLP Engineer, AI Scientist, Machine Learning Engineer

Related on AIssential

• Physics-Guided Policy Optimization with Self-Distillation — PGPO stabilizes LLM self-distillation by modulating step size based on mutual information, inspired by physics.
• When Should the Teacher Move? Temporal Coupling and Stability in Self On-Policy Distillation — Isolation periods, not teacher age, are key for stable self on-policy distillation, preventing state-oblivious collapse.
• A Predictive Law for On-Policy Self-Distillation From World Feedback — On-Policy Self-Distillation (OPSD) performance is predictably linear with the initial student-self-teacher performance gap, enabling pre-training outcome anticipation.
• Theoretical Foundations of Continual Learning via Drift-Plus-Penalty — COLD applies a control-theoretic Drift-Plus-Penalty principle to explicitly regulate forgetting and stability in continual learning systems.
• Extreme Region Policy Distillation — Decoupling RL optimization into aggressive signal extraction and constrained distillation improves both sample and KL efficiency.
• Learning from Your Own Mistakes: Constructing Learnable Micro-Reflective Trajectories for Self-Distillation — TAPO improves LLM self-distillation by explicitly constructing error-specific, natural-language corrective trajectories from contrasting rollouts.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.