SToRe3D: Sparse Token Relevance in ViTs for Efficient Multi-View 3D Object Detection

2026-05-15 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

SToRe3D is a novel relevance-aligned sparsity framework designed to enhance the efficiency of Vision Transformers (ViTs) for multi-view 3D object detection, particularly in autonomous driving. It addresses the high inference latency caused by dense token and query processing across multiple views and large 3D regions. Unlike prior methods that focus on 2D vision or isolated modalities, SToRe3D jointly selects 2D image tokens and 3D object queries, storing filtered features for reactivation. It employs mutual 2D–3D relevance heads to allocate compute to driving-critical content, supervised by a planner-inspired future interaction corridor. Evaluated on nuScenes and the new nuScenes-Relevance benchmark, SToRe3D achieves up to 3x faster inference with marginal accuracy loss, enabling real-time ViT-based 3D detection while maintaining accuracy on planning-critical agents, reaching approximately 18 FPS with a ViT-B backbone.

Key takeaway

For research scientists developing real-time 3D object detection systems for autonomous driving, SToRe3D offers a compelling approach to overcome latency bottlenecks. You should consider implementing joint 2D token and 3D query sparsity, guided by planning-aligned relevance, to achieve significant speedups without sacrificing critical detection accuracy. This method allows for real-time performance on ViT-based architectures, crucial for safety-critical applications.

Key insights

Jointly sparsifying 2D image tokens and 3D object queries with relevance-aligned filtering significantly boosts multi-view 3D detection efficiency.

Principles

• Prioritize compute on planning-critical agents.
• Joint 2D-3D sparsity is more effective than isolated pruning.
• Store-reactivate buffers mitigate information loss from aggressive pruning.

Method

SToRe3D uses mutual 2D–3D relevance heads, supervised by future interaction corridors, to hierarchically filter and store low-relevance tokens/queries, reactivating them at the final layer to preserve context.

In practice

• Implement Gumbel-softmax TopK for differentiable routing.
• Use a linear warm-up schedule for pruning budgets.
• Supervise relevance with a future interaction corridor.

Topics

• Multi-View 3D Object Detection
• Vision Transformers
• Sparse Token Relevance
• Planning-Aligned Perception
• nuScenes-Relevance Benchmark

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Revisiting Token Compression for Accelerating ViT-based Sparse Multi-View 3D Object Detectors — Dynamic patch size adjustment and semantic preservation accelerate ViT-based 3D object detection without accuracy loss.
• COMPOSE: Hypergraph Cover Optimization for Multi-view 3D Human Pose Estimation — Hypergraph partitioning for multi-view 3D pose estimation improves accuracy and robustness over pairwise methods.
• VideoDetective: Clue Hunting via both Extrinsic Query and Intrinsic Relevance for Long Video Understanding — Integrating extrinsic query relevance with intrinsic video structure enhances long video understanding for MLLMs.
• Learned Non-Maximum Suppression for 3D Object Detection — Learned filtering modules, D2D-Rescore and GossipNet3D, enhance 3D object detection by replacing heuristic NMS with detection-to-detection relation leveraging.
• WildDet3D - an open model for monocular 3D detection — WildDet3D predicts 3D object bounding boxes from single RGB images across 13K+ categories without fine-tuning.
• Global-Local Monte Carlo Tree Search in Vision-Language Models for Text-to-3D Indoor Scene Generation — PRM-guided MCTS with hierarchical scene representation enables LVLMs to generate realistic 3D indoor scenes by correcting errors and balancing search.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.