GeoProp: Grounding Robot State in Vision for Generalist Manipulation

2026-07-08 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Robotics & Autonomous Systems, Artificial Intelligence & Machine Learning · Depth: Expert, medium

Summary

GeoProp is a new plug-and-play adapter designed to improve generalist robotic manipulation by explicitly aligning proprioception with visual data. Traditional methods often fail to ground the robot's 3D kinematics within 2D visual feature maps, causing policies to underperform. GeoProp addresses this by projecting the robot's state onto the image plane, sampling localized visual features to create a "grounded state token," and injecting spatial priors into visual features using FiLM modulation. It also incorporates look-ahead visual context by sampling features at a short-horizon predicted coordinate based on recent kinematics. This lightweight adapter, adding only 2-3% to the parameter count, significantly boosts performance: it improves Diffusion Policy by 8.7% on 63 simulation tasks, pi_0 by 4.0% on the RoboTwin subset, and achieves a 10.6% average gain across both policy families in real-world scenarios.

Key takeaway

For Machine Learning Engineers developing generalist robot policies, GeoProp offers a simple yet effective inductive bias to overcome proprioception-vision misalignment. You should consider integrating this lightweight adapter, which adds only 2-3% to parameter count, to significantly improve manipulation performance. This can yield substantial gains, such as 8.7% on simulation tasks and 10.6% in real-world scenarios, enhancing policy grounding and overall robustness.

Key insights

GeoProp aligns robot proprioception with vision through geometric grounding and spatial feature sampling for improved manipulation.

Principles

• Explicitly align 3D kinematics with 2D visual features.
• Inject state-derived spatial priors into visual features.
• Incorporate look-ahead visual context from predicted motion.

Method

GeoProp projects robot state onto the image plane, samples localized visual features for a grounded state token, and injects spatial priors via FiLM modulation. It also samples features at a short-horizon predicted coordinate.

In practice

• Integrate GeoProp to enhance existing diffusion policies.
• Apply GeoProp for improved real-world robot manipulation.
• Use GeoProp to reduce parameter count overhead.

Topics

• Robotic Manipulation
• Proprioception-Vision Fusion
• Generalist Policies
• Diffusion Policy
• FiLM Modulation
• Embodied AI

Code references

• InternRobotics/InternVLA-M1

Best for: Computer Vision Engineer, Research Scientist, AI Scientist, Robotics Engineer, Machine Learning Engineer

Related on AIssential

• GeoProp: Grounding Robot State in Vision for Generalist Manipulation — Explicit geometric grounding of proprioception in vision dramatically improves generalist robot manipulation policies.
• G$^3$VLA: Geometric inductive bias for Vision-Language-Action Models — G$^3$VLA injects calibrated 3D geometry into VLA models' visual streams, improving robot manipulation by resolving 2D-3D coordinate mismatch.
• RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies — RoboDojo unifies sim-and-real evaluation to diagnose generalist robot manipulation policies' capabilities and limitations.
• Proprioceptive-visual correspondence enables self-other distinction in humanoid robots — Proprioceptive-visual correspondence enables robots to distinguish self from others and learn a 3D self-model without prior knowledge.
• Where Should Action Generation Begin? A Learnable Source Prior for Generative Robot Policies — A learnable, proprioception-conditioned prior significantly improves generative robot policy performance and efficiency.
• GroundedPlanBench: Spatially grounded long-horizon task planning for robot manipulation — Jointly planning actions and spatial grounding improves robot task success and action accuracy for VLMs.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.