PGGA: A Plan-Grounded GUI Agent for Automated Device Support

2026-06-25 · Source: Paper Index on ACL Anthology · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, medium

Summary

The Plan-Grounded GUI Agent (PGGA) addresses the limitations of current GUI agents in multi-step digital device support, which often fail due to a procedural knowledge deficit and over-reliance on zero-shot visual exploration. PGGA frames interface navigation as a knowledge-execution problem, conditioning low-level actions on explicit step-by-step text plans. Evaluated on the Device-Support Interaction Benchmark (DSIB), the GTA1-7B model achieved 99.59% Operation Accuracy with expert plans, but only 82.99% Element Accuracy and 45.61% Task Success Rate; without plans, its Task Success Rate was 0.00%. A fine-tuned 2B-parameter PGGA achieved a 54.39% Task Success Rate and 91.28% Element Accuracy when guided by expert plans, demonstrating that explicit procedural grounding significantly enhances GUI execution, provided high-quality plans are available. The project page is https://hsiung.cc/PGGA/ and the paper was presented at ALVR in July 2026.

Key takeaway

For AI scientists and ML engineers developing GUI automation agents, relying solely on visual exploration is insufficient for multi-step tasks. You should integrate explicit procedural grounding by conditioning low-level actions on step-by-step text plans. This approach, demonstrated by PGGA's improved task success and element accuracy, is critical for robust device support. Focus on generating or acquiring high-quality plans to maximize your agent's performance.

Key insights

Explicit procedural grounding via step-by-step text plans substantially improves GUI agent task success and element accuracy.

Principles

• GUI agents need explicit procedural knowledge.
• Visual exploration alone fails multi-step tasks.
• High-quality plans boost GUI execution.

Method

PGGA conditions low-level GUI actions on step-by-step text plans, framing interface navigation as a knowledge-execution problem, contrasting with zero-shot visual exploration.

In practice

• Use text plans for multi-step GUI automation.
• Prioritize element grounding in agent design.
• Develop high-quality procedural instructions.

Topics

• GUI Agents
• Automated Device Support
• Procedural Knowledge
• Language and Vision
• Task Automation
• Human-Computer Interaction

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Robotics Engineer

Related on AIssential

• Workflow-GYM: Towards Long-Horizon Evaluation of Computer-use Agentic tasks in Real-World Professional Fields — Current AI agents struggle with long-horizon, domain-specific GUI workflows, achieving only ~30% success.
• Codex Browser Use IS INSANE! Controls Your Computer & Automates Everything! — Codex, powered by GPT 5.5, offers autonomous AI agent capabilities for software development, testing, and computer control.
• UI-KOBE: Knowledge-Oriented Behavior Exploration for Lightweight Graph-Guided GUI Agents — UI-KOBE improves lightweight GUI agents by guiding them with autonomously constructed, app-specific knowledge graphs, enhancing reliability and privacy.
• Empowering GUI Agents via Autonomous Experience Exploration and Hindsight Experience Utilization for Task Planning — Hindsight experience utilization and autonomous exploration significantly boost planning and generalization in small MLLMs.
• DragOn: A Benchmark and Dataset for Drag-Based GUI Interactions — DragOn addresses the critical gap in drag grounding data for GUI agents, significantly improving VLM performance.
• VISUALSKILL: Multimodal Skills for Computer-Use Agents — Multimodal skills, integrating visuals with text, significantly improve computer-use agents' performance on GUI tasks.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Paper Index on ACL Anthology.