EgoDyn-Bench: Evaluating Ego-Motion Understanding in Vision-Centric Foundation Models for Autonomous Driving

2026-07-08 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Data Science & Analytics · Depth: Expert, extended

Summary

EgoDyn-Bench is a new diagnostic benchmark designed to evaluate the semantic ego-motion understanding of vision-centric foundation models for autonomous driving. It maps continuous vehicle kinematics to discrete motion concepts using a deterministic oracle, effectively decoupling a model's internal physical logic from its visual perception. A large-scale audit of over 20 models, including closed-source MLLMs and open-source VLMs, identified a significant "Perception Bottleneck": models consistently fail to accurately align physical concepts with visual observations, often performing worse than classical geometric baselines. This structural deficit persists across model scales and domain-specific training. The benchmark also revealed that providing explicit trajectory encodings substantially restores physical consistency, indicating that ego-motion logic is derived almost exclusively from the language modality, with visual observations contributing negligible additional signal. The benchmark uses 14,000 QA pairs from 1,000 3-second driving scenarios, combining real-world nuScenes data with augmented CARLA simulations.

Key takeaway

For autonomous driving engineers developing vision-centric foundation models, recognize that current architectures struggle to visually ground ego-motion. Your models will likely derive physical logic from language, not visual observations. To ensure physically consistent behavior, you must either integrate explicit kinematic data as auxiliary input or prioritize pre-training strategies that natively align dynamic representations with visual perception. Relying solely on visual input for ego-motion understanding introduces significant safety risks.

Key insights

Vision-centric foundation models exhibit a structural deficit in grounding ego-motion understanding in visual observations, relying instead on language modality.

Principles

• Current VLM architectures functionally decouple visual perception from physical reasoning.
• Model scaling and domain-specific training do not resolve the visual grounding deficit.
• Explicit kinematic data significantly improves ego-motion reasoning in VLMs.

Method

EgoDyn-Bench formulates ego-motion understanding as a semantic video QA task, using a deterministic oracle to map continuous kinematics to discrete labels for physically-grounded evaluation.

In practice

• Integrate explicit trajectory encodings (e.g., Timeseries) as auxiliary input for VLMs.
• Prioritize native alignment between dynamic representations and visual perception during VLM pre-training.

Topics

• Ego-Motion Understanding
• Vision-Language Models
• Autonomous Driving
• Foundation Models
• Benchmark Evaluation
• Kinematic Reasoning

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Decoding Pedestrian Crossing Intention from Egocentric Vision via Vision Language Models — Vision Language Models, fine-tuned with egocentric and gaze cues, significantly improve pedestrian crossing intention prediction.
• Causality-Aware End-to-End Autonomous Driving via Ego-Centric Joint Scene Modeling — Causality-aware modeling improves end-to-end autonomous driving by capturing ego-agent interdependencies for consistent trajectory prediction.
• Action Draft and Verify: A Self-Verifying Framework for Vision-Language-Action Model — ADV combines diffusion's precision with auto-regression's robustness for VLA models via a draft-and-verify mechanism.
• Motion-Focused Latent Action Enables Cross-Embodiment VLA Training from Human EgoVideos — Unlabeled human videos can effectively train VLA models by disentangling motion and action intent.
• DriveJudge: Rethinking Autonomous Driving Evaluation with Vision-Language Models — DriveJudge combines VLM reasoning with rule-grounded evaluation for interpretable, context-aware autonomous driving assessment.
• VisualThink-VLA: Visual Intermediate Reasoning for Effective and Low-Latency Vision-Language-Action Policies — Visual intermediate reasoning, not textual chain-of-thought, enables effective and low-latency vision-language-action policies.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.