Direct agents with visual prompts in Design Mode

2026-06-05 · Source: Cursor Blog · Field: Technology & Digital — Software Development & Engineering, Artificial Intelligence & Machine Learning · Depth: Intermediate, quick

Summary

Cursor's updated Design Mode introduces a visual prompting interface for AI agents, enabling designers, PMs, and frontend developers to edit product UIs directly within the running application. This feature, accessible via the Cursor browser, allows users to convey intent by clicking specific elements, drawing on the page, or using voice commands. Design Mode provides comprehensive context to agents, including the selected element's code, surrounding layout, visual relationships, and a precise screenshot of the page state. It supports multi-selection for coordinated changes and integrates with Composer 2.5, facilitating rapid, targeted UI modifications and hot reloads. This system streamlines the iteration loop, allowing users to multitask and manage multiple subagents for efficient interface refinement.

Key takeaway

For frontend developers and UI/UX designers iterating on product interfaces, Design Mode offers a significant workflow acceleration. You can now provide AI agents with precise visual context—pointing, drawing, or narrating changes directly on the running application—to achieve faster, more accurate code edits. This reduces the cognitive load of translating visual intent into text prompts, allowing you to maintain flow and rapidly refine interfaces. Consider integrating this visual editing approach to streamline your UI development cycles.

Key insights

Visual prompting in Design Mode enhances AI agent-driven UI editing by providing rich contextual data for faster, more precise iterations.

Principles

• Spatial UI work benefits from visual context.
• Direct manipulation improves agent understanding.
• Fast, targeted models accelerate UI refinement.

Method

Convey intent to agents by clicking elements, multi-selecting, drawing on a frozen viewport frame, or narrating changes via voice input.

In practice

• Edit UI components directly in running app.
• Adjust groups of elements simultaneously.
• Use voice and drawing for complex changes.

Topics

• Design Mode
• Visual Prompting
• AI Agents
• UI Development
• Cursor Browser
• Frontend Tools

Best for: Product Manager, Software Engineer, Product Designer, AI Product Manager

Related on AIssential

• Two users, one CLI: people and agents - Mistral AI — Designing CLIs for AI agents improves tool composability, scriptability, and testability for all users.
• 🔵 Google Transforms the Design Process with "Vibe Design" Agents — AI is rapidly transforming product design, development workflows, and user interaction across desktop and mobile platforms.
• 🔵 Google Search now generates UI on demand — AI is rapidly transforming product development, shifting focus from coding to idea validation and enabling on-demand UI generation.
• VISUALSKILL: Multimodal Skills for Computer-Use Agents — Multimodal skills, integrating visuals with text, significantly improve computer-use agents' performance on GUI tasks.
• Promptions helps make AI prompting more precise with dynamic UI controls — Dynamic prompt refinement controls improve user interaction and reduce effort in generative AI comprehension tasks.
• Coming Soon: Build Interactive Agents with Generative UI — Generative UI allows AI agents to dynamically display custom, interactive components instead of text.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Cursor Blog.