Seeing Together:Multi-Robot Cooperative Egocentric Spatial Reasoning with Multimodal Large Language Models

2026-05-18 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Data Science & Analytics · Depth: Expert, medium

Summary

A new benchmark and framework, CoopSR and SP-CoR, address the challenge of multi-robot cooperative dynamic spatial reasoning using Multimodal Large Language Models (MLLMs). CoopSR introduces EgoTeam, the first dataset for this task, comprising 114,227 QA pairs across 19 question types, four difficulty tiers, and three team sizes in simulated environments (Habitat and iGibson), alongside a real-world test set of approximately 2,326 QAs from two quadruped robots. The proposed SP-CoR framework enhances fine-grained cooperative spatial reasoning by integrating dynamics-aware multi-robot frame sampling, spectral- and physics-guided view fusion, and physics-aligned prompt distillation. This allows SP-CoR to benefit from privileged robot-pose supervision during training while only requiring egocentric videos at test time. SP-CoR consistently outperforms 22 MLLM baselines, achieving a +3.87% improvement on Habitat and +7.12% on iGibson, demonstrating stronger generalization to unseen team sizes and real-world scenarios.

Key takeaway

For research scientists developing multi-robot systems, this work highlights the critical need for specialized datasets and physics-informed MLLM architectures. You should consider integrating dynamics-aware sampling and physics-guided view fusion into your models to enhance cooperative spatial reasoning, especially when aiming for robust generalization across varying team sizes and real-world deployments. The EgoTeam dataset offers a valuable resource for benchmarking and training your next-generation multi-robot perception systems.

Key insights

Multi-robot egocentric spatial reasoning benefits from physics-informed MLLMs and specialized cooperative datasets.

Principles

• Integrate dynamics-aware sampling for multi-robot video.
• Fuse views using spectral and physics guidance.
• Distill prompts with physics alignment.

Method

SP-CoR combines dynamics-aware multi-robot frame sampling, spectral- and physics-guided view fusion, and physics-aligned prompt distillation to enable MLLMs to perform cooperative spatial reasoning from egocentric videos.

In practice

• Utilize EgoTeam dataset for multi-robot QA.
• Apply SP-CoR for improved cooperative reasoning.
• Test generalization with unseen team sizes.

Topics

• Multi-Robot Cooperative Reasoning
• Egocentric Spatial Reasoning
• Multimodal Large Language Models
• CoopSR Benchmark
• EgoTeam Dataset

Code references

• KPeng9510/seeing-together
• zhengxuJosh/Awesome-Spatial-Reasoning
• xqwang14/EVA02-AT

Best for: Computer Vision Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, Robotics Engineer

Related on AIssential

• Seeing Together:Multi-Robot Cooperative Egocentric Spatial Reasoning with Multimodal Large Language Models — MLLMs can achieve cooperative spatial reasoning by integrating synchronized egocentric videos from multiple robots.
• How Does AI Learn to See in 3D and Understand Space? — Bridging 2D AI predictions with 3D geometry via geometric fusion enables scalable, semantic 3D scene understanding.
• Divide, Deliberate, Decide: A Multi-Agent Framework for Fine-Grained Egocentric Action Recognition — Multi-agent VLM deliberation with heterogeneous specialists improves fine-grained egocentric action recognition by decorrelating model priors.
• Keep It in Mind: User Centric Continual Spatial Intelligence Reasoning in Egocentric Video Streams — The DirectMe framework and UCS-Bench dataset advance egocentric AI by enabling robust, user-centric continual spatial reasoning in video streams.
• Towards Learning a Generalizable 3D Scene Representation from 2D Observations — A Generalizable Neural Radiance Field predicts 3D workspace occupancy from robot observations in a global frame.
• Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understanding — Video generation models inherently learn robust 3D structural priors and physical laws for scene understanding.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.