RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation

2026-06-10 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

RLCSD (Reinforcement Learning with Contrastive on-policy Self-Distillation) is a novel method designed to overcome "privilege-induced style drift" in on-policy self-distillation (OPSD). OPSD typically provides dense, token-level supervision by aligning a model's distribution with a privileged context, such as a verified solution. However, this often causes the learning signal to focus on stylistic tokens rather than task-bearing ones, leading to shorter outputs and training instability. RLCSD addresses this by contrasting the teacher-student gap under a correct hint against that under a wrong hint, thereby suppressing style shifts induced by hints and concentrating the signal on task-relevant tokens. Experiments on Qwen3 (1.7B/4B/8B) and Olmo-3-7B-Think models demonstrate RLCSD's consistent outperformance of GRPO and prior OPSD methods across mathematical and logical reasoning tasks. The contrastive principle is also shown to be general, improving existing OPSD methods and extending to cross-model on-policy distillation.

Key takeaway

For Machine Learning Engineers developing reasoning models with on-policy self-distillation, you should consider implementing RLCSD. This method directly addresses "privilege-induced style drift" by using a contrastive approach, ensuring your models learn task-relevant information rather than just stylistic changes from hints. Adopting RLCSD can stabilize training and improve performance on tasks like mathematical and logical reasoning, as demonstrated on Qwen3 and Olmo-3-7B-Think models.

Key insights

RLCSD employs contrastive learning to mitigate "privilege-induced style drift" in on-policy self-distillation, focusing the signal on task-bearing tokens.

Principles

• Hinting can cause "privilege-induced style drift."
• Contrast correct and wrong hint gaps for signal focus.
• Contrastive learning improves on-policy distillation.

Method

RLCSD contrasts the teacher-student gap from correct hints against wrong hints. This suppresses hint-induced style shifts, focusing the learning signal on task-bearing tokens for reasoning models.

In practice

• Integrate contrastive principle into OPSD methods.
• Extend contrastive insight to cross-model distillation.
• Apply RLCSD to mathematical and logical reasoning.

Topics

• RLCSD
• Contrastive Learning
• On-Policy Self-Distillation
• Reasoning Models
• Mathematical Reasoning
• Logical Reasoning

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• Reinforcement Learning from Rich Feedback with Distributional DAgger — DistIL uses a forward cross-entropy objective with rich feedback to guarantee monotonic policy improvement in reinforcement learning.
• Respecting Self-Uncertainty in On-Policy Self-Distillation for Efficient LLM Reasoning — Entropy-guided self-distillation improves LLM reasoning by adaptively weighting teacher supervision based on confidence.
• OPD+: Rethinking the Advantage Design for On-Policy Distillation — Stop-gradient in on-policy distillation biases reward estimates; OPD+ offers a corrected framework for improved teacher-student model transfer.
• Learning from Language Feedback via Variational Policy Distillation — VPD co-evolves teacher and student policies to overcome sparse rewards in RL via dynamic language feedback interpretation.
• Stabilizing On-Policy Distillation for MLLM Reasoning with Global Normalization — GNDPO stabilizes on-policy distillation for MLLM reasoning by normalizing KL scores into batch-level relative advantages.
• On-Policy Context Distillation for Language Models — OPCD enables language models to internalize in-context knowledge by training on self-generated trajectories against a context-conditioned teacher.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.