Extreme Region Policy Distillation

2026-06-06 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, extended

Summary

Extreme Region Policy Distillation (ERPD) is a novel two-stage framework designed to resolve the fundamental trade-off between sample efficiency and asymptotic performance in reinforcement learning for large language models. Traditional off-policy methods often underutilize rich training signals due to conservative optimization, while aggressive updates lead to policy drift and entropy collapse. ERPD addresses this by first performing weakly constrained off-policy optimization on fixed data to extract maximal training signals, creating an "extreme region policy" teacher. In the second stage, these signals are distilled into the base policy under trust-region constraints, filtering harmful drift. Experiments on mathematical reasoning tasks using models like Qwen3-4B and Qwen3.5-27B demonstrate that ERPD achieves comparable or superior performance with significantly smaller KL divergence. The framework also supports "weak-to-strong" distillation, where even degenerate teachers provide effective supervision, and combining signals from multiple teachers further enhances performance.

Key takeaway

For Machine Learning Engineers optimizing LLMs with reinforcement learning, consider implementing Extreme Region Policy Distillation (ERPD) to overcome the sample efficiency-stability trade-off. Your teams can achieve higher performance with less KL divergence by aggressively extracting signals in a first stage and then carefully distilling them. Explore using both strong teachers (e.g., from SAPO/CE) and weak teachers (e.g., MSE-trained with unlearned policy reference) to maximize signal utility, potentially combining them for robust improvements on mathematical reasoning or coding tasks.

Key insights

Decoupling RL optimization into aggressive signal extraction and constrained distillation improves both sample and KL efficiency.

Principles

• Aggressive off-policy updates fully exploit data but cause policy drift.
• Distillation can filter policy drift while preserving performance gains.
• Weaker teachers can provide effective distillation signals.

Method

ERPD uses a two-stage process: first, aggressive, weakly constrained off-policy optimization to create a teacher policy; then, trust-region constrained distillation of its token-level signals into a student policy.

In practice

• Use SAPO or CE for strong teacher training.
• Employ MSE loss for weak teacher signal construction.
• Combine strong and weak teacher signals for robust gains.

Topics

• Reinforcement Learning
• Large Language Models
• Policy Distillation
• Sample Efficiency
• KL Divergence
• Trust Region Methods
• Mathematical Reasoning

Code references

• modelscope/evalscope

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• OPD+: Rethinking the Advantage Design for On-Policy Distillation — Stop-gradient in on-policy distillation biases reward estimates; OPD+ offers a corrected framework for improved teacher-student model transfer.
• Interpretable Policy Distillation for Power Grid Topology Control — Policy distillation can transform complex RL agents into interpretable, high-performing, and deployable surrogates for critical infrastructure control.
• Reinforcement Learning from Rich Feedback with Distributional DAgger — DistIL uses a forward cross-entropy objective with rich feedback to guarantee monotonic policy improvement in reinforcement learning.
• Stabilizing On-Policy Distillation for MLLM Reasoning with Global Normalization — GNDPO stabilizes on-policy distillation for MLLM reasoning by normalizing KL scores into batch-level relative advantages.
• KD-MARL: Resource-Aware Knowledge Distillation in Multi-Agent Reinforcement Learning — KD-MARL distills expert coordination into lightweight, critic-free agents for efficient MARL deployment in resource-constrained settings.
• Learning from Language Feedback via Variational Policy Distillation — VPD co-evolves teacher and student policies to overcome sparse rewards in RL via dynamic language feedback interpretation.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.