VISUALSKILL: Multimodal Skills for Computer-Use Agents

2026-06-16 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Advanced, quick

Summary

VISUALSKILL is a novel hierarchical multimodal skill designed to enhance Computer-Use Agents (CUAs) in handling long-horizon tasks and unfamiliar software. Unlike existing text-only skill libraries, VISUALSKILL integrates both text and visual figures, recognizing the visual nature of GUI interactions. It is tailored to specific applications, organized as a central index over per-topic files, and accessed by agents using a load_topic MCP tool. The skill is constructed via a two-stage pipeline combining authored documentation with live-application UI exploration. On CUA-World and OSExpert-Eval benchmarks, a Claude Code CLI agent backed by Claude Opus 4.6 achieved an average score of 0.456 with VISUALSKILL, representing a +15.3 point absolute lift over a no-skill baseline (0.303). This also marks an +8.3 point gain over a matched text-only skill (0.373), directly demonstrating the benefit of visual figures for UI element identification and workflow state verification.

Key takeaway

For AI Engineers developing computer-use agents for GUI automation, you should integrate multimodal skill artifacts that include visual figures. This approach significantly boosts agent performance, as demonstrated by an +8.3 point gain over text-only skills, by improving UI element identification and workflow state verification. Consider adopting a two-stage skill construction pipeline, combining documentation with live UI exploration, to enhance your agents' adaptability to unseen software and complex tasks.

Key insights

Multimodal skills, integrating visuals with text, significantly improve computer-use agents' performance on GUI tasks.

Principles

• Visual figures aid UI element identification.
• Visuals help verify workflow state post-action.
• Tailor skills to target applications.

Method

Construct skills using a two-stage pipeline: combine authored documentation with live-application UI exploration to generate multimodal artifacts.

In practice

• Implement multimodal skill artifacts for GUI agents.
• Use a load_topic tool for on-demand skill fetching.
• Explore UI directly to enrich skill documentation.

Topics

• Computer-Use Agents
• Multimodal Skills
• GUI Automation
• Large Language Models
• Skill Libraries
• UI Exploration

Code references

• XMHZZ2018/VisualSkills

Best for: Research Scientist, AI Scientist, AI Engineer, Machine Learning Engineer

Related on AIssential

• See, Plan, Snap: Evaluating Multimodal GUI Agents in Scratch — ScratchWorld benchmarks multimodal GUI agents in Scratch, revealing a gap between planning and fine-grained GUI manipulation.
• Seeing Is Not Screening: Multimodal Hidden Instruction Attacks on Agent Skill Scanners — Multimodal hidden instructions exploit text-only agent skill scanners, necessitating execution-grounded multimodal defense.
• New AI agent learns to use CAD to create 3D objects from sketches — An AI agent can learn to operate complex CAD software by observing human UI interactions from a specialized video dataset.
• Teach-and-Repeat: Accurately Extracting Operational Knowledge from Mobile Screen Demonstrations to Empower GUI Agents — Teach VLM accurately extracts step-wise operational knowledge from mobile screen demonstrations, enhancing GUI agent performance.
• bytedance / UI-TARS-desktop — TARS is a multimodal AI agent stack designed for human-like task automation across various interfaces.
• Workflow-GYM: Towards Long-Horizon Evaluation of Computer-use Agentic tasks in Real-World Professional Fields — Current AI agents struggle with long-horizon, domain-specific GUI workflows, achieving only ~30% success.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.