VIA: Visual Interface Agent for Robot Control

2026-07-13 · Source: Artificial Intelligence · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

VIA (Visual Interface Agent for robot control) is a novel framework that redefines robot control as an agentic task, leveraging off-the-shelf foundation models without robot-specific fine-tuning. It enables an FM-powered agent to operate a manipulator via a browser-based 3D interface, perceiving visual input, issuing intuitive commands, observing results, and making adjustments. This approach grants the agent general reasoning, closed-loop error recovery, and planning capabilities from its observations, eliminating the need for privileged state information. VIA successfully solves diverse tabletop manipulation tasks zero-shot using models like Claude Code and Codex. With Fable 5, it achieved 96.7% success on three LIBERO-Goal tasks and 100% on a long-horizon rainbow assembly task, demonstrating that powerful frontier agents can directly control robots through appropriate interfaces.

Key takeaway

For Robotics Engineers developing generalist robot policies, this research suggests you can bypass extensive robot-specific fine-tuning. Instead, consider integrating powerful, off-the-shelf foundation models with visual, browser-based 3D interfaces. This approach allows you to leverage existing FM capabilities for reasoning, planning, and error recovery, potentially accelerating development and achieving high zero-shot success rates on complex manipulation tasks like those demonstrated with Fable 5.

Key insights

General-purpose foundation models can directly control robots through visual interfaces without specialized fine-tuning.

Principles

• Robot control can be reframed as an agentic task.
• Off-the-shelf FMs transfer reasoning to robot control.
• Visual interfaces enable FM-driven manipulator operation.

Method

An FM-powered agent perceives a browser-based 3D interface via screenshots, issues commands, observes outcomes, and adjusts actions for closed-loop control.

In practice

• Employ frontier FMs like Fable 5 for robot tasks.
• Design 3D browser interfaces for agentic robot interaction.
• Achieve zero-shot performance on manipulation tasks.

Topics

• Robot Control
• Foundation Models
• Visual Interface Agent
• Zero-shot Learning
• Robot Manipulation
• Agentic AI

Best for: Research Scientist, AI Scientist, Robotics Engineer, Machine Learning Engineer

Related on AIssential

• Building a Foundation Stack for General-Purpose Robots — X Square Robot's integrated stack for general-purpose robots emphasizes high-quality interaction data, event-based world modeling, and pre-trained, deployable action models.
• VISUALSKILL: Multimodal Skills for Computer-Use Agents — Multimodal skills, integrating visuals with text, significantly improve computer-use agents' performance on GUI tasks.
• Inference-time Policy Steering via Vision and Touch — ViTaL integrates vision and touch for robust robot policy steering in contact-rich manipulation.
• Visually-grounded Humanoid Agents — A two-layer paradigm enables digital human agents to perceive, reason, and act autonomously in novel 3D environments using visual input.
• What matters in building vision–language–action models for generalist robots — RoboVLMs provide a flexible, high-performing framework for generalist robot control by optimizing VLM integration.
• From Web Video to Real-World Robots — Web-scale video pre-training combined with minimal teleoperation data enables highly efficient robot foundation models.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Artificial Intelligence.