DemoPSD: Disagreement-Modulated Policy Self-Distillation

2026-07-02 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

DemoPSD, or Disagreement-Modulated Policy Self-Distillation, is a new framework designed to address critical issues in on-policy self-distillation (OPSD) for training large language models (LLMs) to reason. OPSD, where a single model acts as both teacher and student, often leads to overfitting, suppressed exploration, and "privileged information leakage" due to dense token-level supervision. DemoPSD resolves these by employing "selective adoption of teacher guidance," steering the student towards a reverse-KL barycenter target. This target is a weighted geometric combination of teacher and student distributions, balancing teacher learning with student reasoning capacity. The framework adaptively controls blending at each token position based on distribution discrepancy. DemoPSD provably achieves leakage attenuation and exploration preservation. Experiments on SciKnowEval across four scientific fields demonstrate it outperforms GRPO and SDPO, maintains higher training entropy, and robustly generalizes to out-of-distribution GPQA benchmarks.

Key takeaway

For Machine Learning Engineers developing reasoning LLMs using self-distillation, you should consider implementing disagreement-modulated policy self-distillation. This approach directly addresses privileged information leakage and suppressed exploration, which can hinder cross-domain generalization. By selectively adopting teacher guidance and balancing distributions, your models can achieve superior performance on benchmarks like SciKnowEval and GPQA, ensuring more robust and generalizable reasoning capabilities.

Key insights

DemoPSD mitigates privileged information leakage and preserves exploration in LLM self-distillation via selective teacher guidance.

Principles

• Balance teacher learning with student capacity.
• Discrepancy-based blending improves generalization.
• Avoid dense token-level supervision.

Method

Steer the student toward a reverse-KL barycenter target, a weighted geometric combination of teacher and student distributions, adaptively blending based on token-level distribution discrepancy.

In practice

• Apply reverse-KL barycenter for LLM self-distillation.
• Implement adaptive blending based on teacher-student disagreement.
• Prioritize exploration preservation in reasoning tasks.

Topics

• Policy Self-Distillation
• Large Language Models
• Privileged Information Leakage
• Exploration Preservation
• Machine Learning
• Reasoning LLMs

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Respecting Self-Uncertainty in On-Policy Self-Distillation for Efficient LLM Reasoning — Entropy-guided self-distillation improves LLM reasoning by adaptively weighting teacher supervision based on confidence.
• OmniOPSD: Rationale-Privileged On-Policy Self-Distillation for Affective Computing — OmniOPSD uses rationale-privileged self-distillation to enable MLLMs to learn complex affective tasks without direct imitation or inference-time rationales.
• Learning from Language Feedback via Variational Policy Distillation — VPD co-evolves teacher and student policies to overcome sparse rewards in RL via dynamic language feedback interpretation.
• RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation — RLCSD employs contrastive learning to mitigate "privilege-induced style drift" in on-policy self-distillation, focusing the signal on task-bearing tokens.
• Learning from Your Own Mistakes: Constructing Learnable Micro-Reflective Trajectories for Self-Distillation — TAPO improves LLM self-distillation by explicitly constructing error-specific, natural-language corrective trajectories from contrasting rollouts.
• Reinforcement Learning from Rich Feedback with Distributional DAgger — DistIL uses a forward cross-entropy objective with rich feedback to guarantee monotonic policy improvement in reinforcement learning.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.