TurnOPD: Making On-Policy Distillation Turn-Aware for Efficient Long-Horizon Agent Training

2026-07-07 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

TurnOPD is a novel turn-level budgeting strategy designed to enhance the efficiency of on-policy distillation (OPD) for training long-horizon language agents. It addresses two key inefficiencies in vanilla agent OPD: the waste of wall-clock resources on tail turns providing weak KL supervision during full-horizon rollouts, and the concentration of trajectory-level KL loss on shallow tokens, which leaves deeper decision turns under-trained. TurnOPD introduces two budget controllers: adaptive rollout-depth budgeting, which dynamically determines rollout length using probe-based turn statistics, and progressive turn-normalized loss budgeting, which gradually shifts KL weighting from token-level to turn-balanced supervision. Experiments conducted on ALFWorld, WebShop, and Multi-Hop Search, utilizing task-specialized teacher models, demonstrate that TurnOPD achieves superior validation accuracy within equivalent wall-clock training budgets, significantly advancing the accuracy-time frontier compared to vanilla OPD.

Key takeaway

For Machine Learning Engineers training long-horizon language agents, TurnOPD offers a significant efficiency improvement over vanilla on-policy distillation. You should consider implementing its adaptive rollout-depth and progressive turn-normalized loss budgeting to reduce wall-clock training time while achieving superior validation accuracy. This approach helps you avoid wasting compute on less informative tail turns and ensures deeper decision turns are adequately trained, optimizing your resource allocation for complex agentic tasks.

Key insights

TurnOPD enhances on-policy distillation efficiency for long-horizon agents by introducing turn-aware budgeting strategies.

Principles

• Full-horizon rollouts often yield noisy KL supervision from tail turns.
• Trajectory-level KL objectives can neglect deeper decision-making turns.
• Turn-level budgeting improves on-policy distillation efficiency.

Method

TurnOPD employs adaptive rollout-depth budgeting via probe-based turn statistics and progressive turn-normalized loss budgeting, shifting KL weighting from token-level to turn-balanced supervision.

In practice

• Apply turn-aware budgeting to long-horizon agent training.
• Use probe-based statistics for dynamic rollout depth.
• Shift KL weighting for balanced turn supervision.

Topics

• On-Policy Distillation
• Language Agents
• Long-Horizon Tasks
• Agent Training Efficiency
• Adaptive Rollout Budgeting
• Turn-Aware Learning

Best for: Research Scientist, AI Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agents — Guided On-Policy Distillation improves multi-turn agent capability transfer by curriculum-based teacher intervention, preventing student error compounding.
• PowerOPD: Stabilizing On-Policy Distillation with Bounded Power Transformation — PowerOPD stabilizes on-policy distillation by using bounded power-transformed rewards, significantly improving performance and efficiency.
• Extreme Region Policy Distillation — Decoupling RL optimization into aggressive signal extraction and constrained distillation improves both sample and KL efficiency.
• What and When to Distill: Selective Hindsight Distillation for Multi-Turn Agents — SERL selectively reweights RL objectives using environment feedback to improve credit assignment in multi-turn LLM agents.
• Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models — On-policy distillation (OPD) suffers from length inflation and instability, which StableOPD addresses with divergence constraints and mixture distillation.
• Stabilizing On-Policy Distillation for MLLM Reasoning with Global Normalization — GNDPO stabilizes on-policy distillation for MLLM reasoning by normalizing KL scores into batch-level relative advantages.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.