DocOS: Towards Proactive Document-Guided Actions in GUI Agents

2026-05-18 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

DocOS introduces a novel paradigm called Proactive Document-Guided Action for Graphical User Interface (GUI) agents operating in dynamic, open-web environments. This approach enables agents to autonomously search for and utilize relevant online documentation to resolve long-tailed tasks, addressing a limitation where traditional GUI agents rely solely on static parametric knowledge. To evaluate this capability, DocOS proposes a benchmark that assesses an agent's ability to navigate a web browser, locate documentation, comprehend procedural instructions, and ground them into executable GUI actions. Experiments indicate that current agents face significant challenges in reliably locating relevant information during proactive search and accurately translating retrieved instructions into precise actions, highlighting document-guided interaction as a critical area for developing self-evolving GUI agents.

Key takeaway

For research scientists developing GUI agents, this work highlights that integrating proactive document-guided action is crucial for handling complex, long-tailed tasks. You should focus on improving agents' capabilities in reliably locating relevant online documentation and faithfully grounding retrieved procedural instructions into precise, executable GUI actions to enable more robust and self-evolving systems.

Key insights

GUI agents can overcome long-tailed task limitations by proactively searching and grounding documentation.

Principles

• Static parametric knowledge limits GUI agents.
• Proactive documentation search mirrors human problem-solving.

Method

Agents autonomously navigate web browsers, locate online documentation, comprehend instructions, and ground them into executable GUI actions.

In practice

• Develop agents for proactive documentation search.
• Improve instruction grounding into GUI actions.

Topics

• GUI Agents
• Proactive Document-Guided Action
• DocOS Benchmark
• Procedural Knowledge
• Information Grounding

Best for: Research Scientist, AI Scientist, AI Engineer, Machine Learning Engineer

Related on AIssential

• GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RL — Tailored training and data curation significantly boost GUI agent performance on complex navigation tasks.
• $π$-Bench: Evaluating Proactive Personal Assistant Agents in Long-Horizon Workflows — Proactive assistance in AI agents requires anticipating unstated user needs across long, multi-turn interactions.
• AGORA: An Archive-Grounded Benchmark for Agentic Workplace Document Reasoning — Archive-grounded reasoning in large, messy document collections remains a significant challenge for LLM agents, even with advanced benchmarks.
• AI Agents of the Week: Papers You Should Know About — AI agents are evolving into autonomous operators capable of complex, iterative, and safe real-world task execution.
• What Do Humans Need From Docs? — Human documentation must shift from exhaustive reference to building mental models for effective AI agent interaction.
• Workflow-GYM: Towards Long-Horizon Evaluation of Computer-use Agentic tasks in Real-World Professional Fields — Current AI agents struggle with long-horizon, domain-specific GUI workflows, achieving only ~30% success.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.