Avenir-UX: Automated UX Evaluation via Simulated Human Web Interaction with GUI Grounding

2026-02-10 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Data Science & Analytics · Depth: Expert, extended

Summary

OpenFlo is a new user-experience (UX) evaluation agent that automates web usability testing by simulating human interaction with websites. Built on the Avenir-Web framework (Li et al., 2026), OpenFlo uses GUI grounding to interact with real web pages, unlike traditional tools relying on DOM parsing. The system integrates simulated user behavior profiles with a structured evaluation protocol, including the System Usability Scale (SUS) (Brooke, 1996), step-wise Single Ease Questions (SEQ) (Sauro and Dumas, 2009), and concurrent Think Aloud reasoning. This approach generates comprehensive UX reports with quantitative scores and qualitative insights, identifying specific friction points. A case study on Recreation.gov yielded a SUS score of 55.0/100.0 (Grade D), while a Discogs task achieved an 87.5 SUS score (Grade A+), demonstrating its ability to handle complex web environments and provide actionable feedback.

Key takeaway

For research scientists developing web applications, OpenFlo offers a robust solution for integrating continuous, high-fidelity UX evaluation directly into the software development lifecycle. You should consider adopting OpenFlo to automate usability testing, enabling faster iteration and ensuring your products meet user needs by identifying specific friction points through its combined quantitative and qualitative reporting.

Key insights

OpenFlo automates web UX evaluation by simulating human interaction with GUI grounding and combining quantitative and qualitative metrics.

Principles

• Visual grounding improves web agent robustness.
• Combine quantitative and qualitative UX metrics.
• Automated UX evaluation accelerates development cycles.

Method

OpenFlo employs a multimodal grounding approach, combining DOM parsing with coordinate-based visual tagging. A central MLLM reasons about optimal next steps, while an adaptive memory and dynamic checklist maintain context. The evaluation pipeline includes Think Aloud, step-wise SEQ, and post-task SUS assessments.

In practice

• Use OpenFlo for continuous, scalable UX testing.
• Integrate SUS and SEQ for standardized usability metrics.
• Leverage Think Aloud for qualitative insights into user friction.

Topics

• OpenFlo
• Automated UX Evaluation
• GUI Grounding
• System Usability Scale
• Single Ease Question

Code references

• Onflow-AI/OpenFlo

Best for: Research Scientist, AI Scientist, AI Engineer, MLOps Engineer

Related on AIssential

• Ouvia: A User-centered Framework for Measuring Usability of Speech Translation in Real-World Communication Scenarios — Real-world, user-centered evaluation is crucial for understanding speech translation usability beyond holistic quality.
• DRS-GUI: Dynamic Region Search for Training-Free GUI Grounding — DRS-GUI improves MLLM GUI grounding by dynamically searching relevant screen regions using human-like perceptual actions.
• AMALIA-VL: A Native European Portuguese Open-Source Vision and Language Model — AMALIA-VL is the first open-source LVLM natively optimized for European Portuguese, addressing a critical language resource gap.
• I Gave Qwen3.7-Plus a Screenshot and It Found the Exact Pixel to Click for $0.40 — Qwen3.7-Plus provides precise and affordable GUI grounding, enabling advanced "computer use" agents via a simple API.
• WinDeskGround: A Benchmark for Robust GUI Grounding in Complex Multi-Window Desktop Environments — MLLMs struggle with GUI grounding robustness in complex, multi-window desktop environments, especially under occlusion.
• Article: The AI Productivity Paradox in Test Automation: Moving Beyond Structural Validation to Perception and Intent — Reliable test automation requires validating structure, perception, and business intent, not just DOM.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.