Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretraining

2026-05-14 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Data Science & Analytics · Depth: Expert, quick

Summary

Video2GUI is an automated framework designed to extract grounded GUI interaction trajectories from unlabeled internet videos, addressing the scarcity of large-scale training data for graphical user interface (GUI) agents. The framework utilizes a coarse-to-fine filtering strategy to identify high-quality GUI tutorial videos and convert them into structured agent trajectories. Applying this pipeline to 500 million video metadata entries, the researchers constructed WildGUI, a large-scale dataset comprising 12 million interaction trajectories across more than 1,500 applications and websites. Pre-training models like Qwen2.5-VL and Mimo-VL on WildGUI resulted in consistent performance improvements of 5-20% on various GUI grounding and action benchmarks, achieving or exceeding state-of-the-art results. The WildGUI dataset and Video2GUI pipeline will be released to support future research.

Key takeaway

For research scientists developing GUI agents, the WildGUI dataset and Video2GUI pipeline offer a critical resource for overcoming data scarcity. You should consider integrating WildGUI into your pre-training regimens for multimodal large language models to achieve substantial performance gains, potentially matching or surpassing current state-of-the-art benchmarks.

Key insights

Automated extraction of GUI interaction data from unlabeled videos significantly improves GUI agent pre-training.

Principles

• Unlabeled video data is a rich source for GUI trajectories.
• Coarse-to-fine filtering enhances data quality.

Method

Video2GUI employs a coarse-to-fine filtering strategy to identify high-quality GUI tutorial videos from 500 million metadata entries and converts them into structured interaction trajectories for agent training.

In practice

• Pre-train multimodal LLMs on WildGUI for GUI tasks.
• Utilize Video2GUI for custom GUI data generation.

Topics

• GUI Agents
• Multimodal LLMs
• Video2GUI Framework
• WildGUI Dataset
• Interaction Trajectories

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretraining — Automated extraction of GUI interaction data from unlabeled videos significantly enhances agent pretraining.
• AQuaUI: Visual Token Reduction for GUI Agents with Adaptive Quadtrees — AQuaUI reduces visual tokens for GUI agents by exploiting screenshot spatial redundancy with adaptive quadtrees.
• VideoNet: A Large-Scale Dataset for Domain-Specific Action Recognition — VideoNet introduces a large-scale dataset and benchmark to re-evaluate VLM performance on domain-specific action recognition.
• GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RL — Tailored training and data curation significantly boost GUI agent performance on complex navigation tasks.
• DATAREEL: Automated Data-Driven Video Story Generation with Animations — DataReel benchmark and multi-agent framework automate animated data video generation, outperforming single-prompt LLMs.
• FlexQwen: Exploring Hybrid Objectives and Text Originality for Portuguese — Hybrid objectives and text originality can enhance efficient LLM pre-training for low-resource languages.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.