Improving 3D Labeling in Self-Driving by Inferring Vehicle Information using Vision Language Models

2025-05-21 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

The article presents an approach to improve 3D vehicle labeling in self-driving applications through zero-shot inference of vehicle information. It utilizes a Vision Language Model (VLM) to infer a vehicle's make, model, and generation from image crops, and output accurate 3D bounding box dimensions to seed manual labeling. The method employs iterative prompt engineering, with the "Refined VMMGR Prompt" proving most effective by incorporating occlusion assessment, configuration identification, and modification/damage assessment. Experiments on proprietary data (3,821 vehicle labels) and the Waymo Open Dataset (1,931 vehicle samples) show that advanced VLMs like Gemini Pro 2.5 (with N=10 context images) outperform a strong oracle baseline, achieving the lowest errors and highest IoU. The approach also identifies and corrects suboptimal human labels, particularly in cases of occlusion or when labels include protrusions like side mirrors, demonstrating its potential to reduce manual labeling time and increase quality.

Key takeaway

For Machine Learning Engineers developing autonomous vehicle perception systems, you should integrate Vision Language Models (VLMs) into your 3D auto-labeling pipelines. This approach, particularly with refined VMMGR prompts, can significantly reduce manual labeling time and improve label quality, especially for occluded vehicles or those with subtle modifications. Consider using VLMs like Gemini Pro 2.5 to generate more accurate seed bounding boxes and identify potential human labeling errors.

Key insights

VLMs can significantly enhance 3D vehicle auto-labeling in self-driving by inferring precise dimensions and VMMGR, even correcting human errors.

Principles

• Vehicle Make, Model, Generation Recognition (VMMGR) provides strong priors for 3D dimensions.
• Iterative prompt engineering significantly improves VLM accuracy for specific tasks.
• VLMs can surpass lidar-aided human labels in challenging occlusion scenarios.

Method

Feed 2D image crops of vehicles to a VLM with a refined VMMGR prompt. The VLM assesses occlusion, identifies make, model, generation, configuration, and modifications, then outputs factory 3D dimensions in JSON.

In practice

• Integrate VLM-based dimension inference into existing labeling pipelines.
• Flag modified or damaged vehicles for special labeling focus.
• Use VLM outputs to correct occluded or imprecise human labels.

Topics

• 3D Auto-labeling
• Vision Language Models
• Self-Driving Vehicles
• Vehicle Make Model Recognition
• Prompt Engineering
• Bounding Box Inference
• Waymo Open Dataset

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Zero-Shot Semantic Re-Identification for Autonomous Driving: A VLM Baseline Study — Zero-shot semantic re-identification using VLMs generates textual descriptions for robust, interpretable identity matching in autonomous driving.
• Planning with the Views via Scene Self-Exploration — Self-exploration and view graph distillation enable VLMs to plan multi-turn camera movements in 3D environments.
• VISA: VLM-Guided Instance Semantic Auditing for 3D Occupancy World Models — VISA uses offline VLMs as reliability-aware auditors to improve 3D occupancy world models' semantic accuracy during training.
• Action Draft and Verify: A Self-Verifying Framework for Vision-Language-Action Model — ADV combines diffusion's precision with auto-regression's robustness for VLA models via a draft-and-verify mechanism.
• Occ-VLM: Occupancy Grounded Vision Language Model for Indoor Scene Understanding — Occ-VLM unifies 2D semantics and 3D geometry for scene understanding using only RGB images and occupancy as a spatial prior.
• DriveJudge: Rethinking Autonomous Driving Evaluation with Vision-Language Models — DriveJudge combines VLM reasoning with rule-grounded evaluation for interpretable, context-aware autonomous driving assessment.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.