See, Plan, Snap: Evaluating Multimodal GUI Agents in Scratch

2026-02-11 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Advanced, medium

Summary

ScratchWorld is a new benchmark introduced to evaluate multimodal AI agents' ability to construct programs through Graphical User Interfaces (GUIs) in block-based programming environments like Scratch. Released on February 11, 2026, this benchmark features 83 tasks across four categories: Create, Debug, Extend, and Compute, grounded in the Use-Modify-Create pedagogical framework. It utilizes two interaction modes: a primitive mode for assessing visuomotor control via drag-and-drop, and a composite mode using high-level semantic APIs to isolate program reasoning. An execution-based evaluation protocol validates functional correctness through runtime tests in a browser. Initial experiments with multimodal language models and GUI agents reveal a significant "reasoning--acting gap," indicating challenges in fine-grained GUI manipulation despite strong planning capabilities.

Key takeaway

For research scientists developing multimodal GUI agents, you should prioritize addressing the identified "reasoning--acting gap." Your models may exhibit strong planning, but the benchmark highlights persistent weaknesses in executing precise, fine-grained GUI manipulations. Focus on improving visuomotor control and robust interaction with graphical elements to enhance agent performance in block-based programming environments like Scratch.

Key insights

ScratchWorld benchmarks multimodal GUI agents in Scratch, revealing a gap between planning and fine-grained GUI manipulation.

Principles

• Evaluate visuomotor control separately from program reasoning.
• Validate program correctness via execution-based runtime tests.

Method

ScratchWorld uses 83 tasks in Create, Debug, Extend, and Compute categories, with primitive (drag-and-drop) and composite (semantic API) interaction modes, evaluated by browser-based runtime tests.

In practice

• Test GUI agents with both low-level and high-level interaction modes.
• Focus agent development on fine-grained GUI manipulation.

Topics

• GUI Agents
• Multimodal Language Models
• ScratchWorld Benchmark
• Program Synthesis
• Visuomotor Control

Code references

• sjz5202/GUI-AIMA

Best for: Research Scientist, AI Researcher, AI Scientist, Machine Learning Engineer

Related on AIssential

• PAGER: Bridging the Semantic-Execution Gap in Point-Precise Geometric GUI Control — Precision-sensitive GUI tasks require point-level accuracy and geometry-aware verification, exposing a "Semantic-Execution Gap" in current models.
• VISUALSKILL: Multimodal Skills for Computer-Use Agents — Multimodal skills, integrating visuals with text, significantly improve computer-use agents' performance on GUI tasks.
• MAS-Bench: A Unified Benchmark for Shortcut-Augmented Hybrid Mobile GUI Agents — Hybrid GUI-shortcut agents significantly boost mobile task success and efficiency over GUI-only approaches.
• WatchAct: A Benchmark for Behavior-Grounded Robot Manipulation — Robot manipulation benchmarks must incorporate observed human behavior to evaluate complex reasoning beyond single-image instructions.
• DeskCraft: Benchmarking Desktop Agents on Professional Workflows and Human-in-the-Loop Collaboration — DeskCraft benchmarks desktop agents on complex, human-collaborative professional workflows, revealing current AI limitations in long-horizon task execution.
• MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agents — MementoGUI improves long-horizon GUI agents by learning to selectively manage multimodal memory, preventing information overload and loss.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.