AgentLens: Adaptive Visual Modalities for Human-Agent Interaction in Mobile GUI Agents

2026-04-23 · Source: cs.MA updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Human-Computer Interaction · Depth: Expert, quick

Summary

AgentLens is a novel mobile GUI agent designed to improve human-agent interaction by adaptively employing three visual modalities: Full UI, Partial UI, and GenUI. Submitted on April 22, 2026, this system addresses the limitations of existing mobile agents that force a choice between transparent foreground execution (preventing multitasking) and background execution (lacking visual awareness). Through iterative formative studies, the researchers found that users prefer a hybrid model with just-in-time visual interaction, where the optimal visualization modality varies by task. AgentLens extends standard mobile agents with adaptive communication actions and utilizes Virtual Display to enable background execution with selective visual overlays. A controlled study with 21 participants showed that 85.7% preferred AgentLens, which also achieved a high usability score of 1.94 Overall PSSUQ and an adoption-intent score of 6.43/7.

Key takeaway

For product designers developing mobile GUI agents, AgentLens demonstrates that offering adaptive visual modalities significantly improves user experience and adoption. You should consider implementing a hybrid interaction model that allows for just-in-time visual feedback, dynamically switching between full, partial, or generated UI views based on task context, to balance transparency with multitasking capabilities.

Key insights

Adaptive visual modalities enhance human-agent interaction in mobile GUI agents, balancing transparency and multitasking.

Principles

• User preference for just-in-time visual interaction.
• Optimal visualization modality is task-dependent.

Method

AgentLens extends mobile agents with adaptive communication actions and uses Virtual Display for background execution with selective visual overlays, offering Full UI, Partial UI, and GenUI modalities.

In practice

• Implement adaptive visual feedback for mobile agents.
• Integrate Virtual Display for background UI rendering.

Topics

• Mobile GUI Agents
• Human-Agent Interaction
• Adaptive Visual Modalities
• AgentLens
• Virtual Display

Best for: AI Scientist, Research Scientist, Product Designer

Related on AIssential

• VisualClaw: A Real-Time, Personalized Agent for the Physical World — VisualClaw is a self-evolving multimodal agent that reduces VLM deployment costs and improves accuracy through hybrid encoding and skill evolution.
• From Chat Fatigue to Instant Action // Donné Stevenson — Dynamic, action-oriented UIs overcome "chat fatigue" and boost agent engagement more effectively than open-ended chat.
• Agentic Authoring of Interactive Multiview Visualizations in Genomics — Agentic LLM iteration significantly enhances genomics visualization quality, but complex agent architectures offer no further benefit.
• Shopping Online Is About To Change Forever — Agentic commerce employs AI agents for proactive, personalized product discovery and visual recommendations, streamlining online shopping.
• GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RL — Tailored training and data curation significantly boost GUI agent performance on complex navigation tasks.
• Build Interactive Agents with Generative UI — AI agents are evolving from text-only outputs to generating interactive, custom user interfaces.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.MA updates on arXiv.org.