TurnOPD: Making On-Policy Distillation Turn-Aware for Efficient Long-Horizon Agent Training

2026-07-08 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, short

Summary

TurnOPD is a novel turn-level budgeting strategy designed to enhance the efficiency of on-policy distillation (OPD) for training long-horizon language agents. Submitted on July 7, 2026, this method addresses two key inefficiencies identified in vanilla OPD: the waste of wall-clock resources on tail turns during full-horizon rollouts, which provide weak KL supervision, and the concentration of trajectory-level KL loss on shallow tokens, leading to under-trained deeper decision turns. TurnOPD introduces two budget controllers: adaptive rollout-depth budgeting, which dynamically determines rollout length using probe-based turn statistics, and progressive turn-normalized loss budgeting, which gradually shifts KL weighting from token-level to a more balanced turn-level supervision. Experiments conducted on ALFWorld, WebShop, and Multi-Hop Search benchmarks, utilizing task-specialized teacher models, demonstrate that TurnOPD achieves superior validation accuracy within equivalent wall-clock training budgets, significantly advancing the accuracy-time frontier beyond traditional OPD approaches.

Key takeaway

For Machine Learning Engineers developing language agents for complex, long-horizon tasks, TurnOPD offers a significant efficiency improvement. You should consider integrating TurnOPD's adaptive rollout-depth and progressive turn-normalized loss budgeting strategies. This approach can reduce wasted training resources and ensure deeper decision turns are adequately trained, leading to superior validation accuracy within your existing wall-clock budgets. Evaluate TurnOPD on your specific agentic tasks to optimize training time and performance.

Key insights

TurnOPD efficiently trains long-horizon agents by optimizing on-policy distillation with turn-aware budgeting for better supervision.

Principles

• Vanilla OPD wastes resources on noisy tail turns.
• Trajectory-level KL over-trains shallow tokens.
• Turn-aware budgeting improves long-horizon agent training.

Method

TurnOPD employs adaptive rollout-depth budgeting using probe-based turn statistics and progressive turn-normalized loss budgeting, shifting KL weighting from token-level to turn-balanced supervision for efficient on-policy distillation.

In practice

• Apply TurnOPD to long-horizon agentic tasks.
• Use probe-based turn statistics for rollout depth.
• Implement turn-normalized loss for deeper turns.

Topics

• On-Policy Distillation
• Long-Horizon Agents
• Language Agent Training
• Reinforcement Learning
• Training Efficiency
• Turn-Aware Budgeting

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents — Guided On-Policy Distillation improves multi-turn agent capability transfer by curriculum-based teacher intervention, preventing student error compounding.
• PowerOPD: Stabilizing On-Policy Distillation with Bounded Power Transformation — PowerOPD stabilizes on-policy distillation by using bounded power-transformed rewards, significantly improving performance and efficiency.
• Agents Fail to Reject Stale Memories — LLM efficiency, steerability, and agent memory coherence face fundamental limitations requiring novel architectural solutions.
• Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models — On-policy distillation (OPD) suffers from length inflation and instability, which StableOPD addresses with divergence constraints and mixture distillation.
• Extreme Region Policy Distillation — Decoupling RL optimization into aggressive signal extraction and constrained distillation improves both sample and KL efficiency.
• What and When to Distill: Selective Hindsight Distillation for Multi-Turn Agents — SERL selectively reweights RL objectives using environment feedback to improve credit assignment in multi-turn LLM agents.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.