SWITCH: Benchmarking Modeling and Handling of Tangible Interfaces in Long-horizon Embodied Scenarios

2026-07-08 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

The SWITCH (Semantic World Interface Tasks for Control & Handling) benchmark addresses a critical gap in embodied AI: effective interaction with tangible control interfaces (TCIs) like light switches and appliance panels. Current benchmarks often overlook the need for commonsense/physics reasoning, causal prediction, and outcome verification in situated settings, especially with partial observability and safety implications. SWITCH-Basic, its first iteration, evaluates five key abilities: task-aware VQA, semantic UI grounding, action generation, state-transition prediction, and result verification. It uses egocentric RGB video input across 351 tasks involving 98 real devices. Evaluations show commercial and open LMMMs perform inconsistently, often over-relying on textual cues and under-using visual/video evidence, even on single-step interactions. The benchmark provides data, code, and held-out splits for reproducible evaluation and future contributions.

Key takeaway

For AI Scientists and Machine Learning Engineers developing embodied agents, you must prioritize robust visual grounding and causal reasoning for tangible control interfaces. Current LMMMs over-rely on text and fail on fine-grained visual details, leading to inconsistent performance in real-world interaction tasks. Focus your efforts on improving models' ability to interpret diverse UI layouts and verify action outcomes to ensure reliable and safe agent deployment.

Key insights

Current LMMMs struggle with real-world tangible interface interaction, particularly visual grounding and causal reasoning.

Principles

• TCI interaction requires multi-modal causal reasoning.
• Visual context is under-utilized by LMMMs.
• Fine-grained perception is a bottleneck for action generation.

Method

SWITCH evaluates models via five tasks: VQA, UI grounding, action generation, state prediction, and result verification using egocentric video.

In practice

• Focus LMM training on fine-grained visual perception.
• Develop models for diverse TCI layouts.
• Integrate post-action verification into agent design.

Topics

• Embodied AI
• Tangible Control Interfaces
• Multimodal Models
• Benchmark Evaluation
• Causal Reasoning
• Action Generation
• Visual Question Answering

Code references

• BAAI-Agents/SWITCH

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Robotics Engineer

Related on AIssential

• See, Plan, Snap: Evaluating Multimodal GUI Agents in Scratch — ScratchWorld benchmarks multimodal GUI agents in Scratch, revealing a gap between planning and fine-grained GUI manipulation.
• Physically Native World Models: A Hamiltonian Perspective on Generative World Modeling — Hamiltonian World Models offer a physically grounded framework for embodied AI by integrating energy-based latent dynamics.
• ESI-Bench: Towards Embodied Spatial Intelligence that Closes the Perception-Action Loop — Embodied spatial intelligence requires active perception-action loops, where models currently exhibit "action blindness" and metacognitive gaps.
• From Prompts to Pavement Through Time: Temporal Grounding in Agentic Scene-to-Plan Reasoning — Temporal grounding in AV LLM/LMM ensembles reshapes reasoning but doesn't statistically improve standard correctness metrics, yet shows qualitative benefits.
• TouchThinker: Scaling Tactile Commonsense Reasoning to the Open World with Large-scale Data and Action-aware Representation — TouchThinker scales open-world tactile commonsense reasoning through large-scale data and action-aware representation.
• ADAPT: Benchmarking Commonsense Planning under Unspecified Affordance Constraints — Embodied agents require dynamic affordance reasoning to adapt to unspecified, changing real-world object constraints.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.