SceneGraphGrounder: Zero-Shot 3D Visual Grounding via Structured Scene Graph Matching

2026-05-22 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

SceneGraphGrounder is a novel framework for zero-shot 3D visual grounding, localizing objects in unstructured environments from free-form natural language using only RGB-D inputs. It reformulates 3D grounding as structured graph matching over a reconstructed 3D scene graph. The system introduces a visual marker prompting strategy, enabling a Vision-Language Model (VLM) to infer object-object relationships from 2D views, which are then lifted into a persistent 3D scene graph encoding spatial and semantic relations. Given a query, it constructs a query graph and performs constrained alignment. Experiments on the ScanRefer benchmark show competitive performance among zero-shot approaches, achieving 36.8% Acc@0.25 and 26.0% Acc@0.5. Real-world deployment on a Boston Dynamics Spot robot demonstrated a 51.2% success rate, validating robust spatial reasoning in physical environments.

Key takeaway

For robotics engineers developing autonomous systems requiring robust open-vocabulary 3D visual grounding, SceneGraphGrounder offers a compelling approach. You should consider implementing explicit 3D scene graph representations and graph matching for improved spatial consistency and interpretability, especially for compositional queries. This method's demonstrated real-world performance on a mobile robot suggests it can enhance your system's ability to localize objects from natural language in complex, unstructured environments.

Key insights

Zero-shot 3D visual grounding is achieved by reformulating it as structured graph matching over a reconstructed 3D scene graph.

Principles

• Explicit relational representations improve interpretability.
• Graph matching enhances multi-view consistency.
• VLM-based marker prompting lifts 2D relations to 3D.

Method

SceneGraphGrounder reconstructs 3D scenes from RGB-D, extracts 2D object relationships via VLM marker prompting, lifts these to a 3D scene graph, and performs visual grounding by matching a language query graph to the scene graph. A VLM tie-break module resolves ambiguities.

In practice

• Use RGB-D for 3D scene reconstruction.
• Employ VLM with marker prompting for 2D relation extraction.
• Integrate VLM for ambiguous case tie-breaking.

Topics

• 3D Visual Grounding
• Scene Graphs
• Vision-Language Models
• Graph Matching
• Robotics
• Zero-Shot Learning
• RGB-D Sensing

Code references

• ultralytics/ultralytics

Best for: Computer Vision Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, Robotics Engineer

Related on AIssential

• DGSG-Mind: Dynamic 3D Gaussian Scene Graphs for Long-Term Scene Understanding and Grounding — DGSG-Mind uses hybrid 3D Gaussian scene graphs and an embodied reasoning agent for robust, dynamic, long-term scene understanding.
• Occ-VLM: Occupancy Grounded Vision Language Model for Indoor Scene Understanding — Occ-VLM unifies 2D semantics and 3D geometry for scene understanding using only RGB images and occupancy as a spatial prior.
• Global-Local Monte Carlo Tree Search in Vision-Language Models for Text-to-3D Indoor Scene Generation — PRM-guided MCTS with hierarchical scene representation enables LVLMs to generate realistic 3D indoor scenes by correcting errors and balancing search.
• Reinforcing 3D Understanding in Point-VLMs via Geometric Reward Credit Assignment — Targeted reward assignment and geometric consistency verification significantly improve 3D spatial reasoning in Point-VLMs.
• See-and-Reach: Precise Vision-Language Navigation for UAVs within the Field of View — Isolating target-visible navigation and using dynamic 3D cues improves UAV precision reaching.
• GroundedPlanBench: Spatially grounded long-horizon task planning for robot manipulation — Jointly planning actions and spatial grounding improves robot task success and action accuracy for VLMs.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.