SceneActBench: Can Agents Act on the 3D Scenes They See?

2026-02-05 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

SceneActBench is a new benchmark designed to evaluate vision-language model (VLM) agents' ability to act on complex, multi-object 3D scenes, addressing a gap in existing benchmarks that focus on textual responses or single-object operations. It features a unified agent-environment loop where agents receive PNG images or video frames and manipulate a 3D environment using Blender code. The benchmark comprises five distinct tasks—Layout, Camera, Articulated, Reconstruction, and Dynamic—derived from 210 source instances, resulting in 520 task cases. Evaluations against hidden ground truth use task-specific geometric metrics. Initial testing of eleven proprietary VLM configurations revealed Overall scores ranging from 38.6 to 50.2, with Doubao Seed 2.0 Pro achieving the highest at 50.2. No single VLM consistently excelled across all tasks, highlighting varied strengths and persistent challenges in 3D scene understanding and action.

Key takeaway

For AI Engineers developing or evaluating VLM agents for 3D scene manipulation, recognize that current models lack consistent performance across diverse tasks like layout, articulation, and reconstruction. You should prioritize developing specialized capabilities rather than relying on generalist improvements, as more interaction or visual input does not guarantee better results. Utilize diagnostic metrics to pinpoint specific failure stages, guiding targeted model improvements for robust 3D agent behavior.

Key insights

VLM agents acting on 3D scenes require distinct capabilities, as no single model consistently performs well across diverse tasks.

Principles

• 3D scene action demands multiple, specialized capabilities from VLM agents.
• Increased visual input does not guarantee improved performance across all configurations or tasks.
• Higher interaction volume does not correlate with better overall performance in 3D scene manipulation.

Method

SceneActBench evaluates VLM agents by having them observe visual inputs, execute Blender code via a shared Model Context Protocol (MCP) interface, render, and iteratively revise actions against hidden 3D ground truth.

In practice

• Benchmark VLM agents on 3D tasks like layout, camera pose, articulation, reconstruction, and dynamics.
• Employ fine-grained diagnostic metrics to identify specific failure modes in 3D agent performance.

Topics

• Vision-Language Models
• 3D Scene Understanding
• Agent Benchmarking
• Blender Automation
• Geometric Metrics
• Multi-Object Interaction

Code references

• Feinaldo2/SceneActBench

Best for: Research Scientist, Computer Vision Engineer, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• SceneActBench: Can Agents Act on the 3D Scenes They See? — SceneActBench evaluates VLM agents' action capabilities on complex 3D scenes, revealing current models struggle with consistent performance.
• AI agents create virtual playgrounds to help robots get crucial training data — Collaborative AI agents powered by advanced VLMs can autonomously generate highly realistic and diverse 3D simulation environments for robot training, as demonstrated by the "SceneSmith" system.
• 3DCodeBench: Benchmarking Agentic Procedural 3D Modeling Via Code — 3DCodeBench benchmarks VLM agents for procedural 3D modeling, revealing API mismatch issues and the need for better data and iterative refinement.
• Skill-3D: Evolving Scene-Aware Skills for Agentic 3D Spatial Reasoning — MLLM agents improve 3D spatial reasoning by dynamically learning and applying scene-aware tool-use skills.
• MultiView-Bench: A Diagnostic Benchmark for World-Centric Multi-View Integration in VLMs — MultiView-Bench reveals VLMs struggle with world-centric 3D spatial reasoning and multi-view integration, despite 2D proficiency.
• Can Agents Read the Room? Benchmarking Visual Social Intelligence in Multimodal Simulation — Multimodal agents struggle with complex visual social cues for interaction regulation and outcome achievement.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.