D3VL: Understanding Driving Scenes from 3D Time Series Data and Video with Language Models

2026-07-23 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

D3VL is a novel Multimodal Large Language Model (MLLM) framework designed to enhance autonomous driving scene understanding by integrating 2D video and 3D time-series data from LiDAR and stereo cameras. Addressing the challenge of sparse 3D data integration, D3VL improves MLLM accuracy by 11% on the KITTI Question-Answering (QA) dataset, achieving 85% overall accuracy compared to baselines. The framework also introduces the Waymo QA dataset extension, comprising 60K Video-Question-Answer triplets across 768 training and 150 testing scenes, each 20 seconds long. This new dataset assesses models under diverse driving conditions, including adverse weather and nighttime, and focuses on 3D and temporal perception for AV safety.

Key takeaway

For Machine Learning Engineers developing autonomous driving MLLMs, recognize that integrating 3D time-series data is crucial for robust scene understanding. Your current MLLMs likely underperform with 3D inputs; therefore, you should adopt frameworks like D3VL that explicitly process 2D video and 3D sensor data. Utilize comprehensive datasets such as WaymoQA to fine-tune your models, ensuring improved perception of critical objects like cyclists and pedestrians across diverse driving conditions.

Key insights

D3VL enables MLLMs to effectively integrate 2D video and 3D time-series data for enhanced autonomous driving scene understanding.

Principles

• MLLMs require specialized frameworks to effectively process 3D time-series data from LiDAR and stereo cameras.
• Fine-tuning MLLMs on combined 2D and 3D time-series data significantly improves perception of small objects and prominent 3D features.

Method

Convert 3D LiDAR point clouds or stereo images into depth images, tokenize 2D and depth image sequences, project tokens to language space, concatenate with text, and feed into the LLM. Fine-tune for 1-3 epochs.

In practice

• Create depth images from LiDAR or stereo camera data as 3D input for MLLMs.
• Employ semi-supervised annotation for large-scale AV datasets, using a finetuned MLLM for training scenes.

Topics

• D3VL
• Multimodal LLMs
• Autonomous Driving
• 3D Time-Series Data
• LiDAR Perception
• Waymo QA Dataset
• Scene Understanding

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• D3VL: Understanding Driving Scenes from 3D Time Series Data and Video with Language Models — D3VL integrates 2D/3D time-series data for autonomous driving scene understanding, improving performance over 2D-only MLLMs.
• Improving 3D Labeling in Self-Driving by Inferring Vehicle Information using Vision Language Models — VLMs can significantly enhance 3D vehicle auto-labeling in self-driving by inferring precise dimensions and VMMGR, even correcting human errors.
• Learning Geometric Representations from Videos for Spatial Intelligent Multimodal Large Language Models — GeoVR fundamentally restructures MLLM's internal representations for 3D spatial intelligence using 2D video and 3D foundation model distillation, without inference overhead.
• Stream3D-VLM: Online 3D Spatial Understanding with Incremental Geometry Priors — Stream3D-VLM enables real-time 3D spatial understanding from video streams by integrating incremental geometry and efficient voxel compression.
• Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understanding — Video generation models inherently learn robust 3D structural priors and physical laws for scene understanding.
• 3D-Aware VLMs with Implicit and Explicit Geometries — VLM-IE3D improves vision-language models' 3D spatial reasoning by integrating implicit and explicit geometries from RGB video.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.