Learning Geometric Representations from Videos for Spatial Intelligent Multimodal Large Language Models

2026-06-06 · Source: cs.AI updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

Multimodal Large Language Models (MLLMs) excel at 2D semantic understanding but lack intrinsic 3D awareness, resulting in representations that fail to maintain geometric and spatial consistency across video frames. GeoVR, a novel framework, learns geometric representations using purely 2D video sequences, enabling spatial intelligence by restructuring MLLM's semantic latent space. It distills geometry knowledge from pre-trained 3D foundation models through a multi-objective learning strategy. This strategy uses four geometric targets: inter-frame camera pose estimation, dense depth map regression, metric scale factor prediction, and multi-scale 3D feature distillation. Experiments on spatial reasoning benchmarks like VSI-Bench show GeoVR-2B achieves a 69.1 average score, outperforming Qwen3-VL-2B-Instruct (50.3) by 18.8 points and surpassing proprietary models like GPT-5 (55.0) and other 3D-aware models like Cambrian-S-7B (67.5) without additional inference overhead.

Key takeaway

For AI Scientists and ML Engineers developing MLLMs for spatial reasoning, GeoVR offers a critical paradigm shift. You can now instill robust 3D awareness into your models using readily available 2D video data, bypassing the limitations of scarce 3D datasets. This method avoids inference overhead, making it practical for deployment. Consider integrating multi-objective geometric learning to fundamentally restructure your MLLM's latent space for superior spatial intelligence.

Key insights

GeoVR fundamentally restructures MLLM's internal representations for 3D spatial intelligence using 2D video and 3D foundation model distillation, without inference overhead.

Principles

• MLLMs lack intrinsic 3D awareness from 2D pre-training.
• Distilling 3D priors into MLLM latent space enhances spatial intelligence.
• Multi-objective geometric learning from 2D video bypasses 3D data scarcity.

Method

GeoVR employs a multi-objective learning strategy during training, using a frozen 3D foundation model as a teacher to provide pseudo-labels for camera poses, depth maps, metric scale, and multi-scale geometric feature alignment. Auxiliary heads are discarded at inference.

In practice

• Enhance MLLMs for physical world reasoning.
• Improve spatial understanding in dynamic video scenarios.
• Develop 3D-aware models without explicit 3D datasets.

Topics

• Multimodal Large Language Models
• Geometric Representation Learning
• 3D Foundation Models
• Spatial Reasoning
• Video Understanding
• Depth Estimation

Code references

• WHB139426/GeoVR-MLLM

Best for: Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• Learning Geometric Representations from Videos for Spatial Intelligent Multimodal Large Language Models — GeoVR enhances MLLMs' 3D spatial intelligence by distilling geometric knowledge from 2D videos using multi-objective learning.
• Learning Geometric Representations from Videos for Spatial Intelligent Multimodal Large Language Models — GeoVR enables MLLMs to learn 3D spatial intelligence from 2D videos by distilling geometric knowledge via multi-objective learning.
• LongSpace: Exploring Long-Horizon Spatial Memory from Perception to Recall in Video — Long-horizon spatial memory in MLLMs requires integrating 3D structural cues and hierarchical, question-guided memory retrieval.
• Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understanding — Video generation models implicitly learn robust 3D structural priors, which can enhance MLLM spatial reasoning.
• Seeing Together:Multi-Robot Cooperative Egocentric Spatial Reasoning with Multimodal Large Language Models — MLLMs can achieve cooperative spatial reasoning by integrating synchronized egocentric videos from multiple robots.
• How Does AI Learn to See in 3D and Understand Space? — Bridging 2D AI predictions with 3D geometry via geometric fusion enables scalable, semantic 3D scene understanding.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.AI updates on arXiv.org.