Usability Analysis of Configurator User Interfaces with Multimodal Large Language Models

2026-06-04 · Source: cs.SE updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Emerging Technologies & Innovation · Depth: Expert, extended

Summary

A study investigates the use of multimodal large language models (MLLMs) for semi-automated usability analysis of configurator user interfaces (UIs). Researchers synthesized 18 configurator-specific usability criteria from literature and applied them in an MLLM-based analysis of 16 real-world configurators. Using Google's gemini-2.5-flash model and screen recordings, the approach generated severity ratings and improvement suggestions. The MLLM-based analysis averaged 14.4 seconds per criterion and 258.4 seconds per configurator. A review by six software engineers found 88.5% of issue descriptions and 98.0% of improvement recommendations plausible, confirming MLLMs can reliably identify issues and provide domain-aware suggestions, significantly reducing analysis effort.

Key takeaway

For software engineers developing or maintaining complex configurator UIs, this research suggests integrating MLLM-based analysis into your development workflow. You can significantly reduce manual effort in identifying usability issues by leveraging tools like gemini-2.5-flash with screen recordings and domain-specific criteria. While human validation is still necessary, this approach provides rapid, actionable feedback, accelerating iteration cycles and improving user experience, especially for overlooked aspects like variant comparison.

Key insights

MLLMs can semi-automate configurator UI usability analysis using domain-specific criteria and video input, reducing manual effort.

Principles

• Configurator usability requires domain-specific criteria beyond general heuristics.
• MLLMs can interpret visual and textual context for UI analysis.
• Human validation remains crucial for MLLM-generated usability assessments.

Method

Provide an MLLM (e.g., gemini-2.5-flash) with screen recordings of user interactions, 18 configurator-specific usability criteria, and detailed prompts to generate severity ratings and improvement suggestions.

In practice

• Use MLLMs to identify overlooked UI aspects like variant comparison.
• Implement suggested refinements like tooltips or clearer progress indicators.
• Refine MLLM prompts with few-shot examples for better accuracy.

Topics

• Multimodal Large Language Models
• Usability Analysis
• Configurator User Interfaces
• Human-Computer Interaction
• Software Product Lines
• UI Design

Best for: AI Scientist, Research Scientist, Software Engineer

Related on AIssential

• Training Computer Use Agents to Assess the Usability of Graphical User Interfaces — A new CUA, uxCUA, accurately assesses GUI usability and generates critiques by learning from a large, defect-augmented UI dataset.
• Visual Reasoning Benchmark: Evaluating Multimodal LLMs on Classroom-Authentic Visual Problems from Primary Education — MLLMs struggle with dynamic visual reasoning despite proficiency in static visual tasks, posing risks for educational applications.
• From Feelings to Metrics: Understanding and Formalizing How Users Vibe-Test LLMs — Formalized "vibe-testing" bridges the gap between LLM benchmark scores and real-world user utility.
• My unsupervised elicitation challenge — AI models can struggle with specific, low-resource tasks even with general domain knowledge.
• AtelierEval: Agentic Evaluation of Humans & LLMs as Text-to-Image Prompters — AtelierEval benchmarks T2I prompter proficiency using an agentic evaluator, revealing mimicry's superiority over planning.
• Mapping the Design Space of User Experience for Computer Use Agents — Understanding user experience is crucial for designing effective LLM-based computer use agents.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.SE updates on arXiv.org.