Which Way Did It Move? Diagnosing and Overcoming Directional Motion Blindness in Video-LLMs

2026-05-21 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

Video Large Language Models (Video-LLMs) exhibit "directional motion blindness," performing near chance on basic signed image-plane motion direction tasks. This failure stems from a "direction binding gap," where motion information is present in the vision encoder and LLM hidden states but not correctly associated with verbal answers. While synthetic instruction tuning offers limited improvement, researchers introduce MoDirect, a new dataset family, and DeltaDirect, a projector-level objective. DeltaDirect predicts normalized 2-D motion vectors from adjacent-frame feature deltas. On MoDirect-SynBench, DeltaDirect boosts motion direction accuracy from 25.9% to 85.4%. It also improves real-world accuracy on MoDirect-RealBench by 21.9 points without real-world tuning data, all while maintaining standard video-understanding performance.

Key takeaway

For Computer Vision Engineers developing Video-LLMs, addressing fundamental motion direction understanding is crucial. Your models likely suffer from "directional motion blindness" due to a binding gap, even if motion signals are present. Consider integrating the DeltaDirect objective, which significantly improves motion direction accuracy on both synthetic and real-world data without sacrificing general video understanding. This approach offers a targeted solution for enhancing perceptual primitives in your models.

Key insights

Video-LLMs struggle with basic motion direction due to a binding gap, addressable by a novel feature delta objective.

Principles

• Motion direction signal is often present but unbound.
• Visual complexity weakens motion direction signals.
• Targeted projector-level objectives can resolve binding gaps.

Method

DeltaDirect is a projector-level objective predicting normalized 2-D motion vectors from adjacent-frame feature deltas, designed to overcome the direction binding gap in Video-LLMs.

In practice

• Use MoDirect for motion direction tuning and evaluation.
• Implement DeltaDirect to improve Video-LLM motion understanding.
• Focus on projector-level interventions for binding issues.

Topics

• Video-LLMs
• Motion Direction
• Directional Motion Blindness
• DeltaDirect
• MoDirect Dataset
• Computer Vision

Code references

• KHU-VLL/DeltaDirect

Best for: Research Scientist, AI Scientist, Computer Vision Engineer

Related on AIssential

• MotionEnhancer: Leveraging Video Diffusion for Motion-Enhanced Vision-Language Models — MotionEnhancer improves VLM motion understanding by aligning VDM-derived motion priors via parameter-free attention modules.
• Large Depth Completion Model from Sparse Observations — LDCM achieves robust, metric-accurate dense depth completion from sparse observations via a transformer, novel initialization, and 3D coordinate regression.
• Understanding Identity Continuity in Thermal Video through Scene-Level Consistency — High-precision trajectory relinking using scene-level consistency effectively recovers identity in thermal video MOT.
• DIMOS: Disentangling Instance-level Moving Object Segmentation — Disentangling appearance and motion features within and across modalities significantly improves moving instance segmentation, especially for small objects.
• Making Foresight Actionable: Repurposing Representation Alignment in World Action Models — Aligning visual representations with action-grounded semantics improves robot manipulation policies in World Action Models.
• Video Reconstruction using Diffusion-based Image-to-Video Generation with Trajectory Guidance — Trajectory-guided diffusion models can reconstruct missing video frames by integrating external sensor data.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.