AVI-HT: Adaptive Vision-IMU Fusion for 3D Hand Tracking

2026-05-22 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Computer Vision · Depth: Expert, extended

Summary

AVI-HT is an adaptive visual-IMU fusion system designed for 3D hand tracking, combining egocentric images with on-glove 6-DoF IMU signals. Developed by Meta Reality Labs, AVI-HT significantly enhances tracking accuracy and availability, particularly in hand-object interaction scenarios where visual occlusion is prevalent. Its success stems from a synchronized multi-modal training dataset, DexGloveHOI, comprising over 100K vision-IMU samples with ground-truth 3D hand poses, and a novel cross-sensor deep attention mechanism that dynamically adjusts reliance on vision or individual IMU sensors. Experiments on DexGloveHOI demonstrated AVI-HT reduced mean keypoint error by 16.1% and its wrist-aligned variant by 24.2% compared to vision-only baselines like UMETrack, and improved MANO model performance by 23.5% PA-MPJPE. Ablation studies confirmed IMU sensors primarily benefit their respective fingers and provide the greatest gains in highly occluded activities.

Key takeaway

For machine learning engineers developing 3D hand tracking for AR/VR or robotics, you should integrate egocentric vision with on-glove 6-DoF IMU data. This fusion, particularly with an adaptive cross-sensor attention mechanism, dramatically improves accuracy during heavy occlusion, a common challenge in hand-object interactions. Consider building multi-modal datasets with synchronized vision-IMU and ground truth to train robust systems, ensuring reliable performance in complex real-world scenarios.

Key insights

Adaptive fusion of egocentric vision and on-glove 6-DoF IMUs, guided by cross-sensor attention, significantly improves 3D hand tracking under occlusion.

Principles

• Vision and 6-DoF IMUs are complementary for hand tracking.
• Kinematic priors improve multi-modal sensor fusion.
• Adaptive attention dynamically balances sensor reliance.

Method

AVI-HT uses a hierarchical cross-sensor attention module to fuse global visual tokens with 12 per-sensor IMU tokens, guided by a kinematic prior mask, then re-calibrates with a second-level self-attention.

In practice

• Use 6-DoF IMUs for occlusion-robust finger tracking.
• Synchronize multi-modal data for optimal fusion.
• Design attention mechanisms with anatomical priors.

Topics

• 3D Hand Tracking
• Vision-IMU Fusion
• Cross-Sensor Attention
• DexGloveHOI Dataset
• AR/VR Applications
• Robotics Teleoperation

Best for: Computer Vision Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, Robotics Engineer

Related on AIssential

• EgoPressDiff: Multimodal Video Diffusion for Egocentric UV-Domain Hand-Pressure Estimation — EgoPressDiff uses multimodal video diffusion and specialized feature fusion to accurately estimate egocentric hand-pressure maps.
• Seeing Without Eyes: 4D Human-Scene Understanding from Wearable IMUs — 4D human-scene understanding is achievable using only wearable IMUs, bypassing visual perception limitations.
• UMI-3D: Extending Universal Manipulation Interface from Vision-Limited to 3D Spatial Perception — UMI-3D enhances robot manipulation data collection via LiDAR-centric SLAM and multimodal sensor fusion.
• SkillSpotter: Pose-Aware Multi-View Skilled Action Detection and Grading in Ego-Exo Videos — SkillSpotter jointly detects and grades skilled actions in ego-exo videos by fusing visual and pose data across multiple views.
• Efficient Sensor Fusion for Gesture Recognition on Resource-Constrained Devices — Sensor fusion of low-resolution ToF and IR enables efficient, private gesture recognition on resource-constrained devices.
• Multi-modality Image Fusion under Adverse Weather: Mask-Guided Feature Restoration and Interaction — Mask-guided feature restoration and interaction improves multi-modality image fusion under adverse weather.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.