Expanding Spatial and Temporal Context for Robotic Imitation Learning With Scene Graphs

2026-06-08 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

This paper introduces a novel approach for robotic imitation learning in partially observed, large-scale environments by leveraging dynamically updated task-relevant scene graphs. This method addresses challenges in long-horizon tasks and partial observability, where traditional policies struggle with extended spatial and temporal contexts. The scene graph acts as a structured memory, capturing object-centric relationships and their evolution. Experiments on simulated mobile manipulation with a Boston Dynamics Spot robot (400 demonstrations) and real-world tabletop manipulation using a 7-DoF Franka Emika Panda arm (300 demonstrations) demonstrate substantial improvements in policy performance. The approach particularly excels in tasks requiring long-term reasoning and robust generalization, achieving real-time closed-loop control at 3.5-3.8 Hz on a single NVIDIA 3090 GPU.

Key takeaway

For Robotics Engineers developing autonomous agents for complex, long-horizon tasks in partially observed environments, consider integrating dynamic, task-relevant scene graphs into your imitation learning pipelines. This approach, demonstrated to improve performance in mobile and tabletop manipulation, provides explicit memory for object localization and tracking, mitigating error propagation and enhancing generalization. You should explore using LLMs for object enumeration and vision foundation models for scene graph construction to boost policy robustness.

Key insights

Scene graphs provide explicit, compact memory for robots to reason over long-horizon tasks under partial observability.

Principles

• Explicit memory improves long-horizon robotic task success.
• Task-relevant object-centric representations are efficient.
• Spatial information (3D centroids) is crucial for object tracking.

Method

A large language model identifies task-relevant objects from natural language. Grounding DINO detects objects, XMem tracks them, forming nodes with visual embeddings, 2D bounding boxes, and 3D centroids. This scene graph conditions a transformer-based diffusion policy.

In practice

• Use LLMs (e.g., GPT-4) for task-relevant object enumeration.
• Integrate Grounding DINO and XMem for dynamic scene graph construction.
• Condition diffusion policies with structured scene graph representations.

Topics

• Robotic Imitation Learning
• Scene Graphs
• Partial Observability
• Long-Horizon Tasks
• Mobile Manipulation
• Diffusion Policy

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Robotics Engineer

Related on AIssential

• Learning Structural Latent Points for Efficient Visual Representations in Robotic Manipulation — Hybrid structural latent points improve robotic manipulation by combining implicit expressiveness with explicit structural priors.
• Physical Intelligence Team Unveils MEM for Robots: A Multi-Scale Memory System Giving Gemma 3-4B VLAs 15-Minute Context for Complex Tasks — MEM provides VLA models with dual-track memory for long-horizon robotic tasks, enabling adaptation and real-time performance.
• MatterDoor: Sampling Zero-shot Spatio-semantic Priors using Generative Models — Generative models can sample spatio-semantic priors for robotic planning in partially observed 3D environments.
• Multi Graph Search for High-Dimensional Robot Motion Planning — Multi-Graph Search (MGS) offers a complete and bounded-suboptimal approach for high-dimensional robot motion planning.
• v305: Proceedings of CoRL 2025 — Advancements in robot learning focus on generalization, safety, and complex interaction through large models and multi-modal sensing.
• Chameleon: Control-Indexed Prospective Memory for Visuomotor Manipulation — Bio-inspired episodic memory with geometry-grounded encoding and goal-directed recall improves robotic manipulation under perceptual aliasing.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.