When Context Returns: Toward Robust Internalization in On-Policy Distillation

2026-06-10 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

Recent research identifies a critical issue in on-policy distillation: while internalizing privileged context improves a student model's no-context performance, reintroducing that context can paradoxically degrade its accuracy, a phenomenon termed "context-induced degradation." This work argues that robust internalization requires not only matching the teacher's context-conditioned behavior but also maintaining stability when context is reintroduced, a property called "context removability." To address this, a lightweight consistency regularizer is proposed. This method anchors the student's no-context output via stop-gradient and penalizes deviations from context-conditioned output using forward KL divergence, requiring only one extra forward pass per training step. Across 12 configurations spanning diverse domains and model families, the method improves context-conditioned accuracy in most settings, reduces context-induced harm in 11 out of 12 settings, and eliminates response-length inflation.

Key takeaway

For Machine Learning Engineers deploying on-policy distilled models, be aware that reintroducing original context can degrade performance, even on previously correct instances. You should consider implementing the proposed lightweight consistency regularizer, which uses a stop-gradient anchored no-context output and forward KL divergence penalty. This approach effectively mitigates context-induced degradation and improves stability across diverse model families, ensuring robust internalization without performance penalties.

Key insights

Reintroducing privileged context to on-policy distilled models can degrade performance, a problem addressed by a new consistency regularizer ensuring context removability.

Principles

• Context-induced degradation is a key challenge in distillation.
• Robust internalization demands context removability.
• Consistency regularization stabilizes distilled model behavior.

Method

A lightweight consistency regularizer anchors the student's no-context output via stop-gradient, then penalizes context-conditioned output deviation using forward KL divergence, requiring one extra forward pass.

In practice

• Mitigate performance degradation from re-introduced context.
• Improve no-context performance in distilled models.
• Eliminate response-length inflation in student models.

Topics

• On-Policy Distillation
• Context Removability
• Consistency Regularization
• Student Models
• Machine Learning Robustness
• KL Divergence

Best for: Research Scientist, AI Engineer, NLP Engineer, AI Scientist, Machine Learning Engineer

Related on AIssential

• OPRD: On-Policy Representation Distillation — OPRD improves on-policy distillation by aligning hidden states, eliminating sampling variance and providing richer structural information.
• Harmonizing Multi-Objective LLM Unlearning via Unified Domain Representation and Bidirectional Logit Distillation — Multi-objective LLM unlearning requires harmonizing data representation and employing bidirectional distillation for balanced efficacy and robustness.
• MixSD: Mixed Contextual Self-Distillation for Knowledge Injection — Aligning supervision targets with an LLM's native generation distribution significantly mitigates catastrophic forgetting during knowledge injection.
• Escaping the KL Agreement Trap in On-Policy Distillation — Detecting and terminating low-KL agreement traps in on-policy distillation improves training efficiency and accuracy.
• AI 101: "On-Policy Distillation Zeitgeist" — Self-distillation offers a scalable, on-policy method for LLMs to refine reasoning by self-critique and dense feedback.
• Distribution Corrected Offline Data Distillation for Large Language Models — Offline distillation can be improved by correcting teacher-student distribution drift without costly online sampling.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.