UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generation

2026-07-08 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, extended

Summary

UI2App is introduced as the first benchmark specifically designed to evaluate "interaction inference" in executable web application generation from image-only UI screenshots. Comprising 327 screenshots organized into 45 state-coherent sets for multi-route web applications, UI2App assesses generated artifacts across four dimensions: executability, navigation reachability, visual fidelity, and interaction inference (IIS). Experiments with six frontier vision-language models revealed a significant capability mismatch; the visual-fidelity leader scored only 7.5 on IIS, ranking fourth and trailing the IIS leader by 5.2x. High-complexity interactions, particularly cross-page state management, remain a pervasive bottleneck, with half of the models scoring exactly zero in this dimension. Overall, the benchmark highlights that inferring complete interaction behavior from static screenshots is a major challenge for current models.

Key takeaway

For Machine Learning Engineers developing or evaluating vision-language models for UI-to-code generation, you must move beyond visual fidelity metrics. Your models should be rigorously tested for interaction inference, particularly for complex cross-route state management, which is a frontier-wide challenge. Prioritize improving functional correctness and state-management complexity to build truly dynamic and usable web applications, rather than just pixel-perfect static facades.

Key insights

Interaction inference from static UI screenshots is a distinct, challenging capability for VLMs, separate from visual fidelity.

Principles

• Visual fidelity does not imply interaction inference capability in web application generation.
• Cross-route state persistence is a critical and widespread bottleneck for current VLMs.
• Admin applications consistently present the highest interaction inference difficulty for models.

Method

UI2App employs an end-to-end pipeline evaluating executability, navigation, visual fidelity, and interaction inference (IIS) via a rubric-based taxonomy across seven interaction categories and three state-management scopes.

In practice

• Prioritize interaction inference capabilities over visual fidelity in VLM development.
• Focus VLM training on complex state management, especially cross-route persistence.
• Implement robust self-debug mechanisms to address scaffold-respect and factuality errors.

Topics

• Web Application Generation
• Vision-Language Models
• Interaction Inference
• UI-to-Code Benchmarking
• Cross-Route State
• Code Generation Evaluation

Code references

• chenmancm169/UI2App

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• VISTA: An End-to-End Benchmark for Visual Spec-to-Web-App Coding Agents — A new benchmark, VISTA, evaluates LLM agents' end-to-end web-app generation, revealing decoupled visual fidelity and functional correctness.
• Arena-T2I Hard: Benchmarking and Improving Faithfulness with Dependency-Aware Checklist — Complex T2I prompt faithfulness requires dependency-aware evaluation and a balanced reward system to avoid aesthetic bias.
• Cookie-Bench: Continuous On-screen Key Interaction Evaluation for Web Generation — A new evaluation regime offers reference-free, autonomous, and holistic assessment for LLM-generated interactive web applications.
• DiffSpot: Can VLMs Spot Fine-Grained Visual Differences in Web Interfaces? — VLMs struggle with fine-grained visual difference detection in web UIs, achieving only 40.7% accuracy on a new benchmark.
• A Study of Failure Modes in Two-Stage Human-Object Interaction Detection — HOI models struggle with multi-person scenes and object-conditioned verb biases, leading to persistent, high-confidence errors.
• VisualThink-VLA: Visual Intermediate Reasoning for Effective and Low-Latency Vision-Language-Action Policies — Visual intermediate reasoning, not textual chain-of-thought, enables effective and low-latency vision-language-action policies.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.