Adaptive Cross-Modal Fusion with Sparse Attention for Pedestrian Crossing Intention Prediction

2026-07-14 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

ADAPT (Adaptive Domain-Aware Pedestrian Crossing Transformer) is a novel multimodal framework designed for accurate pedestrian crossing intention prediction, a critical task for autonomous driving. It addresses limitations of existing single-modal or dense fusion methods by jointly modeling local and global visual context with temporal motion dynamics. ADAPT integrates four spatially aligned visual modalities—RGB images, local depth maps, global semantic maps, and global depth maps—alongside ego-vehicle speed, pedestrian bounding boxes, and skeleton pose information. The framework employs five specialized modules: a weight-shared Swin Transformer V2 backbone, a Cross-Modality Guided Attention module, a Mamba-based Motion Feature Encoding module, a Sparse Cross-Modal Attention module, and a Vision Transformer-based Temporal Feature Fusion module. Extensive experiments on the JAAD and PIE benchmark datasets show ADAPT outperforms current methods, achieving an AUC of 0.73 on JAADbeh and 0.85 on JAADall, and an accuracy of 0.92 with an AUC of 0.90 on PIE. It also offers real-time efficiency, performing inference in 17.23 ms per sample.

Key takeaway

For autonomous driving engineers developing pedestrian prediction systems, ADAPT offers a robust framework to enhance safety and real-time performance. You should consider its adaptive cross-modal fusion and sparse attention mechanisms to improve prediction accuracy while maintaining low computational overhead. This approach allows for more reliable decision-making in complex urban environments, directly impacting vehicle safety and operational efficiency.

Key insights

ADAPT uses sparse cross-modal attention and multimodal fusion for efficient, accurate pedestrian intention prediction.

Principles

• Combine diverse visual and kinematic data.
• Use sparse attention for inter-modal interactions.
• Model local, global, and temporal contexts.

Method

ADAPT processes four visual modalities and kinematic data through specialized modules for feature extraction, hierarchical visual fusion, Mamba-based temporal encoding, sparse cross-modal attention, and sequence-level prediction.

In practice

• Integrate RGB, depth, and semantic maps.
• Incorporate ego-vehicle speed and skeleton pose.
• Apply Mamba for temporal motion encoding.

Topics

• Pedestrian Crossing Prediction
• Autonomous Driving
• Multimodal Fusion
• Sparse Attention
• Swin Transformer
• Mamba Architecture

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Decoding Pedestrian Crossing Intention from Egocentric Vision via Vision Language Models — Vision Language Models, fine-tuned with egocentric and gaze cues, significantly improve pedestrian crossing intention prediction.
• CityGen: Structure-Guided City-Style Synthesis for Cross-City Autonomous Driving — CityGen enables zero-label cross-city adaptation for autonomous driving via HD-map-conditioned diffusion synthesis, improving robustness.
• TSCA-Net: Temporal-Spatial Clique Attention for Interpretable Multimodal Pedestrian Trajectory Prediction — TSCA-Net improves pedestrian trajectory prediction by dynamically adapting to temporal, spatial, and motion complexity using three specialized modules.
• ROSA-RL: Uncertainty-Aware Roundabout Optimized Speed Advisory with Reinforcement Learning — Probabilistic conflict forecasting with RL enables safe, efficient, uncertainty-aware automated driving in mixed-traffic roundabouts.
• CLEAR: Cognition and Latent Evaluation for Adaptive Routing in End-to-End Autonomous Driving — CLEAR achieves efficient, multi-modal autonomous driving by combining single-step generative planning with LLM-driven cognitive reasoning.
• Multi-Modal Hyper-Graph Fusion for Low-Light Crowd Counting — Multi-modal hyper-graph fusion and sparse attention enhance crowd counting robustness in low-light conditions.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.