UCOB: Learning to Utilize and Evolve Agentic Skills via Credit-Aware On-Policy Bidirectional Self-Distillation

2026-07-20 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, extended

Summary

UCOB, a new framework for agentic reinforcement learning, addresses the fragility of treating skill-conditioned prompts as fixed, privileged teachers. Introduced on June 28, 2026, UCOB employs credit-aware on-policy bidirectional self-distillation (CBSD), which compares skill-conditioned and no-skill prompts based on their return-to-go within the same task and anchor state. The view yielding a higher return is dynamically selected as the local teacher, enabling the internalization of useful skill-conditioned behavior and the correction of misleading skill usage. UCOB also guides task/state skill memory updates, utility-aware retrieval, and reflection self-training. Experiments on ALFWorld, WebShop, and Search-QA demonstrate UCOB's superior performance, achieving up to 23.5 and 18.0 point gains over SOTA baselines on ALFWorld and WebShop, particularly with the Qwen3-1.7B model.

Key takeaway

For AI Engineers developing language agents for multi-turn tasks, you should reconsider fixed skill-conditioned supervision. UCOB's credit-aware bidirectional self-distillation offers a robust approach by dynamically selecting the better-performing context view as the teacher. This method improves agent performance and memory evolution, especially for compact models like Qwen3-1.7B, by internalizing useful skills and correcting misleading ones. Consider integrating this dynamic teacher selection into your agent training pipelines.

Key insights

Skill-conditioned prompts are unreliable teachers; local credit assignment determines distillation direction.

Principles

• Skill-conditioned views are not universally authoritative.
• Teacher direction should be locally value-dependent.
• Memory evolution improves agent performance.

Method

UCOB compares skill-conditioned and no-skill rollouts by return-to-go within anchor states, selecting the higher-return view as the local teacher for bidirectional self-distillation.

In practice

• Implement bidirectional self-distillation for skill learning.
• Group rollouts by anchor state for local credit assignment.
• Dynamically update task and state-level skill memories.

Topics

• Agentic Reinforcement Learning
• Skill Memory
• Self-Distillation
• Language Agents
• Credit Assignment
• Multi-turn Tasks
• Qwen3-1.7B

Code references

• TU2021/UCOB

Best for: NLP Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning — Self-evolving on-policy distillation converts hindsight skills from trajectories into dense token-level supervision for agentic RL.
• How to Fix Your Agent's Amnesia: Lessons from Building a Self-learning Agent — Learning agents, unlike stateless ones, continuously adapt and improve performance by retaining and applying knowledge from past interactions.
• Policy-Conditioned Counterfactual Credit for Verifiable Reinforcement Learning of Long-Horizon Language Agents — Policy-conditioned counterfactual credit and validity gating enable verifiable, reliable long-horizon reinforcement learning for language agents.
• TRIAGE: Role-Typed Credit Assignment for Agentic Reinforcement Learning — TRIAGE improves agentic RL by assigning credit based on semantic action roles, correcting GRPO's uniform outcome-only approach.
• Scaffolding vs Real Intelligence for AI Agents — AI agent learning can shift from reward functions to guided self-correction through Socratic dialogue and diagnostic critique.
• OPID: On-Policy Skill Distillation for Agentic Reinforcement Learning — OPID leverages hierarchical on-policy skill distillation to provide dense, distribution-matched supervision for agentic reinforcement learning, improving performance and efficiency.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.