On-Policy Context Distillation for Language Models

2026-02-12 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

On-Policy Context Distillation (OPCD) is a new framework designed to help language models internalize in-context knowledge directly into their parameters. OPCD achieves this by training a student model on its own generated trajectories, while simultaneously minimizing the reverse Kullback-Leibler divergence against a teacher model conditioned on specific contexts. The framework has been successfully applied to two key areas: experiential knowledge distillation, where models consolidate knowledge from past solution traces, and system prompt distillation, allowing models to internalize behaviors from optimized prompts. Across diverse tasks such as mathematical reasoning, text-based games, and domain-specific applications, OPCD consistently surpasses baseline methods in task accuracy and maintains out-of-distribution capabilities. It also facilitates effective cross-size distillation, enabling smaller student models to learn from larger teachers.

Key takeaway

For AI Engineers developing more efficient and capable language models, OPCD offers a robust method to internalize contextual knowledge and prompt behaviors directly into model parameters. You should consider integrating OPCD to improve task accuracy and preserve out-of-distribution performance, especially when distilling knowledge from larger models to smaller, more deployable student models.

Key insights

OPCD enables language models to internalize in-context knowledge by training on self-generated trajectories against a context-conditioned teacher.

Principles

• Distill knowledge from historical solution traces.
• Internalize behaviors from optimized system prompts.

Method

Train a student model on its own generated trajectories, minimizing reverse Kullback-Leibler divergence against a context-conditioned teacher model.

In practice

• Apply to mathematical reasoning tasks.
• Use for text-based game agents.
• Enable cross-size model distillation.

Topics

• On-Policy Context Distillation
• Language Models
• Knowledge Distillation
• Experiential Knowledge
• System Prompt Distillation

Best for: AI Engineer, Machine Learning Engineer, Research Scientist, AI Researcher, AI Scientist, NLP Engineer

Related on AIssential

• Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe — OPD success hinges on compatible thinking patterns and genuinely new teacher knowledge, not just higher scores.
• OPID: On-Policy Skill Distillation for Agentic Reinforcement Learning — OPID leverages hierarchical on-policy skill distillation to provide dense, distribution-matched supervision for agentic reinforcement learning, improving performance and efficiency.
• Breaking the Tokenizer Barrier: On-Policy Distillation across Model Families — Cross-tokenizer On-Policy Distillation (OPD) enables knowledge transfer between LLMs with different tokenizers via a precise token-mapping algorithm.
• Bridging Reasoning Trajectories in On-Policy Distillation via Near-Future Guidance — On-Policy Distillation's token-level learning struggles with reasoning divergence; near-future trajectory guidance improves alignment and performance.
• Skill-Guided Continuation Distillation for GUI Agents — SGCD closes the supervision gap for GUI agents in off-trajectory states by generating and distilling skill-guided continuations.
• OPD-Evolver: Cultivating Holistic Agent Evolver via On-Policy Distillation — OPD-Evolver uses slow-fast co-evolution and on-policy self-distillation to create agents that holistically manage and learn from experience.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.