DRS-GUI: Dynamic Region Search for Training-Free GUI Grounding

2026-05-18 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

DRS-GUI is a training-free framework designed to improve GUI grounding for Multimodal Large Language Models (MLLMs) by dynamically searching for instruction-relevant regions in high-resolution screenshots. It introduces a lightweight UI Perceptor that executes three human-like perceptual actions: Focus, Shift, and Scatter, to progressively explore the interface and generate region proposals. An Action Planner, based on Monte Carlo Tree Search (MCTS), dynamically schedules these actions, guided by a region quality reward that evaluates and selects highly instruction-relevant areas, pruning redundant UI elements. Experiments on ScreenSpot-Pro, ScreenSpot-V1, and ScreenSpot-V2 benchmarks demonstrate that DRS-GUI yields significant improvements, including a 14% accuracy gain on ScreenSpot-Pro for MLLMs like Qwen2.5-VL-7B and UGround-V1-7B, without requiring additional model training or fine-tuning.

Key takeaway

For research scientists developing GUI agents, DRS-GUI offers a robust, training-free method to significantly improve grounding accuracy, especially on complex, high-resolution interfaces. You should consider integrating this dynamic region search framework into your MLLM pipelines to mitigate visual clutter and enhance localization stability without the need for extensive model retraining or fine-tuning, thereby making your agents more reliable and interpretable.

Key insights

DRS-GUI enhances MLLM GUI grounding via a training-free dynamic region search and MCTS-based action planning.

Principles

• Adaptive perceptual scope improves grounding.
• Step-wise region evaluation prevents error accumulation.
• Interactive elements are better grounding targets.

Method

DRS-GUI uses a UI Perceptor for Focus, Shift, and Scatter actions, scheduled by an MCTS Action Planner, guided by a composite region quality reward (relevance, coverage, concentration) to identify optimal grounding regions.

In practice

• Integrate DRS-GUI as a plug-and-play MLLM enhancement.
• Use MCTS with a budget of N=8 iterations for efficiency.
• Prioritize interactive elements in region relevance scoring.

Topics

• GUI Grounding
• Dynamic Region Search
• Multimodal Large Language Models
• Monte Carlo Tree Search
• UI Perceptor

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• DRS-GUI: Dynamic Region Search for Training-Free GUI Grounding — DRS-GUI improves MLLM GUI grounding by dynamically searching relevant screen regions using human-like perceptual actions.
• WinDeskGround: A Benchmark for Robust GUI Grounding in Complex Multi-Window Desktop Environments — MLLMs struggle with GUI grounding robustness in complex, multi-window desktop environments, especially under occlusion.
• Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Grounding — GUI-SD improves GUI grounding via on-policy self-distillation with visual context and entropy-guided token weighting.
• PruneGround: Plug-and-play Spatial Pruning for 3D Visual Grounding — PruneGround improves 3D Visual Grounding by spatially pruning scenes and refining language descriptions for efficient, accurate object localization.
• Focus When Necessary: Adaptive Routing and Collaborative Grounding for Training-Free Visual Grounding — LazyMCoT adaptively routes visual grounding tasks, combining MLLM attention with external visual expertise for fine-grained detail.
• PointVG-R: Internalizing Geometric Reasoning in MLLMs for Precise Pointing Localization via Visual Chain of Thought — PointVG-R internalizes geometric reasoning in MLLMs for precise pointing localization using visual Chain-of-Thought and adaptive reinforcement learning.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.