Skill-Guided Continuation Distillation for GUI Agents

2026-06-17 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

Skill-Guided Continuation Distillation (SGCD) is an iterative self-improvement framework designed to enhance GUI agents by addressing the "supervision gap" in off-trajectory states. Traditional behavior cloning struggles when an agent's policy deviates from expert trajectories, encountering states without expert demonstrations. SGCD tackles this by first allowing a plain policy to reach these realistic off-trajectory states. Subsequently, a skill-guided policy takes over to complete the task, generating successful continuations. These generated continuations are then combined with original expert trajectories, providing crucial supervision for previously unseen, policy-induced states. The framework extracts "skills" from both successful and failed rollouts, comprising "Continuation Plans," "Critical Targets," "Failure Traps," and "Success Criteria." On the OSWorld-Verified benchmark, SGCD significantly improved the success rate of three base models, elevating performance from the low-30% range to over 50%.

Key takeaway

For Machine Learning Engineers developing GUI automation agents, SGCD offers a robust approach to overcome limitations of pure behavior cloning. If your agents struggle with off-trajectory states or policy drift, consider implementing SGCD's iterative self-improvement. This framework can significantly boost success rates, as demonstrated by improving performance from the low-30% range to over 50% on OSWorld-Verified, by generating crucial supervision for unseen states.

Key insights

SGCD closes the supervision gap for GUI agents in off-trajectory states by generating and distilling skill-guided continuations.

Principles

• Policy deviation creates unseen states lacking supervision.
• Self-improvement can generate supervision for off-trajectory states.
• Skills can be extracted from both success and failure.

Method

SGCD runs a plain policy to off-trajectory states, then a skill-guided policy completes the task, generating continuations. These are mixed with expert data for distillation.

In practice

• Improve GUI agent robustness to policy drift.
• Extract skills like "Failure Traps" from rollouts.
• Enhance success rates on complex GUI tasks.

Topics

• GUI Agents
• Behavior Cloning
• Skill Distillation
• Off-trajectory Learning
• Self-Improvement Frameworks
• OSWorld-Verified Benchmark

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Robotics Engineer

Related on AIssential

• Keep Policy Gradient in Charge: Sibling-Guided Credit Distillation for Long-Horizon Tool-Use Agents — Direct self-distillation can degrade tool-use by misaligning with verifier rewards; distillation should guide credit, not replace policy gradient.
• OS-Themis: A Scalable Critic Framework for Generalist GUI Rewards — OS-Themis is a scalable multi-agent critic framework that improves GUI agent RL training through milestone-based reward decomposition and strict evidence auditing.
• On-Policy Context Distillation for Language Models — OPCD enables language models to internalize in-context knowledge by training on self-generated trajectories against a context-conditioned teacher.
• SkillPyramid: A Hierarchical Skill Consolidation Framework for Self-Evolving Agents — SkillPyramid enables AI agents to dynamically evolve and generalize skills through hierarchical consolidation and self-evolution, improving performance.
• Learning Visual Spatial Planning from Symbolic State via Modality-Gap-Aware Self-Distillation — Visual spatial planning benefits from self-distillation that bridges the perception-reasoning modality gap.
• Learning from Your Own Mistakes: Constructing Learnable Micro-Reflective Trajectories for Self-Distillation — TAPO improves LLM self-distillation by explicitly constructing error-specific, natural-language corrective trajectories from contrasting rollouts.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.