Mapping the Design Space of User Experience for Computer Use Agents

2026-02-12 · Source: Apple Machine Learning Research · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering, Emerging Technologies & Innovation · Depth: Intermediate, quick

Summary

A two-phase study explored the user experience (UX) design space for large language model (LLM)-based computer use agents, which execute user commands via UI interaction. Phase 1 involved reviewing existing systems and conducting interviews with eight UX and AI practitioners to develop a taxonomy of UX considerations, including user prompts, explainability, user control, and users' mental models. Phase 2 utilized a Wizard-of-Oz study with 20 participants, where a researcher simulated a web-based computer use agent, observing user reactions during normal, error-prone, and risky task execution. The study validated the taxonomy and deepened understanding of design connections and diverse user needs, providing a map for developers to consider various UX aspects in agent design.

Key takeaway

For AI Product Managers developing LLM-based computer use agents, understanding the nuanced user experience is critical for adoption. Your design decisions should explicitly address user prompts, explainability, and control, as these factors significantly influence user satisfaction and trust. Prioritize validating your agent's UX against diverse user needs and scenarios to ensure practical utility and mitigate potential frustrations.

Key insights

Understanding user experience is crucial for designing effective LLM-based computer use agents.

Principles

• UX taxonomy guides agent design.
• User control impacts agent adoption.
• Explainability builds user trust.

Method

A two-phase study combined taxonomy development via expert interviews with empirical validation through a Wizard-of-Oz study, probing user reactions to LLM agent interactions.

In practice

• Consider user prompts in agent design.
• Implement explainability features.
• Prioritize user control mechanisms.

Topics

• LLM-based Agents
• User Experience Design
• Computer Use Agents
• Human-AI Interaction
• Wizard-of-Oz Study

Best for: AI Scientist, AI Product Manager, Product Manager, AI Engineer, Product Designer, Research Scientist

Related on AIssential

• Quoting Timothy B. Lee — The effective use of LLMs requires skill development and a learning curve, akin to effective management.
• Does The Way You Plan Matter? An Empirical Study of Planning Representations for LLM Web Agents — Plan representation and LLM choice critically influence LLM web agent robustness and task success.
• PerceptUI: LLM Agents as Human-Aligned Synthetic Users for UI/UX Evaluation — PerceptUI leverages LLM agents to simulate persona-conditioned UI/UX responses and rationales, achieving human-level realism and generalization.
• Beyond expert users: agents should help users construct preferences, not just elicit them — Agents should help users construct preferences by providing domain knowledge, not just elicit pre-formed ones.
• UXBench: Measuring the Actionability of LLM-Generated UX Critiques — LLM UX critique actionability varies significantly, requiring interaction-grounded evaluation beyond static analysis.
• LLM-Enhanced Messaging: The Rise of Agentic Inboxes — LLMs are transforming messaging from simple filtering to agentic inboxes, significantly boosting knowledge worker productivity.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Apple Machine Learning Research.