VLRC: Vision-Language Reprojection Consistency as a scalable signal for better feed-forward 3D pretraining

2026-07-08 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

Vision-Language Reprojection Consistency (VLRC) is introduced as a scalable auxiliary objective for feed-forward 3D models, addressing limitations of expensive geometric supervision and ambiguous photometric self-supervision. VLRC leverages frozen vision-language representations as semantic multi-view supervision. It operates by reprojecting dense vision-language features across views from a predicted 3D reconstruction and enforcing feature consistency at corresponding image locations, requiring no additional 3D annotations. This objective integrates with self-supervised monocular reconstruction (e.g., SS3D) and supervised-pretrained models (e.g., VGGT) during unlabeled adaptation. Experiments on indoor and outdoor benchmarks, including KITTI and NYUv2 for depth, and ScanNet200 for segmentation, demonstrate consistent gains in 3D reconstruction accuracy, depth, camera estimation, and improved zero-shot open-vocabulary 3D semantic segmentation.

Key takeaway

For Machine Learning Engineers developing 3D reconstruction systems, integrating Vision-Language Reprojection Consistency (VLRC) into your training pipeline can significantly improve geometric accuracy and semantic understanding. You should consider applying VLRC as an auxiliary objective to enhance depth and camera estimation, especially when working with self-supervised or unlabeled adaptation scenarios. This approach will lead to more robust 3D models that better support open-vocabulary semantic segmentation.

Key insights

VLRC uses vision-language feature consistency across reprojected views as a scalable signal to improve 3D geometry and semantic alignment.

Principles

• Inconsistent vision-language features across views signal misaligned 3D geometry.
• Aligning geometry with language-grounded features enhances 3D semantic understanding.

Method

VLRC reprojects dense vision-language features from a source view to a target view via predicted 3D structure. It then enforces cosine similarity between these reprojected features and the target view's features.

In practice

• Improve depth and camera pose estimation in 3D models.
• Enable coherent multi-view semantic fusion for open-vocabulary 3D scene understanding.

Topics

• 3D Reconstruction
• Vision-Language Models
• Self-Supervised Learning
• Semantic Segmentation
• Depth Estimation
• Camera Pose Estimation
• Multi-View Geometry

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Reinforcing 3D Understanding in Point-VLMs via Geometric Reward Credit Assignment — Targeted reward assignment and geometric consistency verification significantly improve 3D spatial reasoning in Point-VLMs.
• Anchored, Not Graded: Vision-Language Models Fail at Slant-from-Texture Perception — VLMs fail to express continuous geometric perception due to a language readout bottleneck, despite encoding the information.
• Learning Geometric Representations from Videos for Spatial Intelligent Multimodal Large Language Models — GeoVR enhances MLLMs' 3D spatial intelligence by distilling geometric knowledge from 2D videos using multi-objective learning.
• Judging to Improve: A De-biased VLM-as-3D-Judge Protocol for Single-Image 3D Generation — A hardened VLM-as-3D-judge protocol can specialize 3D generators, but exceeding strong baselines requires more than lightweight PEFT.
• Occ-VLM: Occupancy Grounded Vision Language Model for Indoor Scene Understanding — Occ-VLM unifies 2D semantics and 3D geometry for scene understanding using only RGB images and occupancy as a spatial prior.
• 3D Consistency Optimization for Self-Supervised Monocular Video Depth Estimation — Recasting monocular video depth as multi-view 3D reconstruction with 3D consistency optimization improves geometric accuracy and coherence.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.