SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning

2026-07-16 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

The SEED (SElf-Evolving On-Policy Distillation) framework, published on 2026-07-16, addresses the supervision gap in outcome-based reinforcement learning for large language models acting as interactive agents. This method tackles the challenge of sparse trajectory-level rewards, which offer limited guidance for intermediate decisions in long-horizon tasks. SEED operates by fine-tuning the policy to analyze completed on-policy trajectories, generating natural-language skills that capture reusable workflows or failure-avoidance rules. The current policy both collects trajectories and extracts these hindsight skills, allowing the supervision to evolve alongside policy improvements. It then re-scores sampled actions, converting the skill-induced probability shift into a dense token-level on-policy distillation signal. This signal is jointly optimized with outcome-based RL, ensuring alignment with the current trajectory distribution. Experiments on text-based and vision-based agentic tasks demonstrate SEED's consistent improvements in performance and sample efficiency, alongside robust generalization.

Key takeaway

For Machine Learning Engineers developing interactive LLM agents struggling with sparse rewards in long-horizon tasks, consider implementing SEED's self-evolving on-policy distillation. This approach allows your policy to generate and learn from hindsight skills, providing dense token-level supervision that significantly boosts performance and sample efficiency. You can achieve robust generalization across text-based and vision-based agentic tasks by jointly optimizing this skill-induced signal with outcome-based RL.

Key insights

Self-evolving on-policy distillation converts hindsight skills from trajectories into dense token-level supervision for agentic RL.

Principles

• Policy and skill analysis co-evolve through iterative updates.
• Auxiliary supervision must align with the current trajectory distribution.

Method

Fine-tune the policy to analyze completed trajectories, generate natural-language skills, then re-score actions to create a dense token-level distillation signal jointly optimized with outcome-based RL.

In practice

• Apply to text-based and vision-based agentic tasks.
• Enhance sample efficiency in long-horizon RL.

Topics

• Agentic Reinforcement Learning
• Large Language Models
• On-Policy Distillation
• Hindsight Skills
• Reward Sparsity
• Text-Vision Agents

Code references

• jinyangwu/SEED

Best for: Research Scientist, AI Engineer, Computer Vision Engineer, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• OPID: On-Policy Skill Distillation for Agentic Reinforcement Learning — OPID leverages hierarchical on-policy skill distillation to provide dense, distribution-matched supervision for agentic reinforcement learning, improving performance and efficiency.
• Learning from Your Own Mistakes: Constructing Learnable Micro-Reflective Trajectories for Self-Distillation — TAPO improves LLM self-distillation by explicitly constructing error-specific, natural-language corrective trajectories from contrasting rollouts.
• Training Agents 2: Live tutorial on model distillation for training custom agents. — Model distillation transfers capabilities from large teacher models to smaller student models using various on-policy, off-policy, and self-distillation techniques.
• OPD-Evolver: Cultivating Holistic Agent Evolver via On-Policy Distillation — OPD-Evolver uses slow-fast co-evolution and on-policy self-distillation to create agents that holistically manage and learn from experience.
• Self-Distilled Agentic Reinforcement Learning — SDAR combines gated self-distillation with reinforcement learning to stabilize and enhance LLM agent training.
• UCOB: Learning to Utilize and Evolve Agentic Skills via Credit-Aware On-Policy Bidirectional Self-Distillation — Skill-conditioned prompts are unreliable teachers; local credit assignment determines distillation direction.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.