Multi-THuMBS: Multi-person Tracking of 3D Human Meshes Beyond Video Shots

2026-07-02 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Computer Vision · Depth: Expert, quick

Summary

Multi-THuMBS (Multi-person Tracking of 3D Human Meshes Beyond Video Shots) is a novel approach designed to overcome the challenges of tracking multiple people's 3D human meshes in unconstrained, in-the-wild videos, particularly across frequent shot changes. Existing methods struggle with identity loss and inconsistent trajectories during abrupt camera viewpoint transitions, and are often limited to single-person scenarios. Multi-THuMBS addresses this by leveraging a state-of-the-art 3D scene prior to reconstruct the two boundary frames of a shot change within a single shared 3D space. This allows for the registration of human meshes, preserving per-person identity and motion consistency across these transitions. Extensive experiments demonstrate that Multi-THuMBS significantly improves 3D human mesh recovery, camera pose estimation, and identity tracking compared to previous state-of-the-art methods, ensuring high-fidelity motion reconstruction.

Key takeaway

For Computer Vision Engineers developing multi-person 3D tracking systems for unconstrained, in-the-wild videos, Multi-THuMBS offers a robust solution to a critical challenge. Its ability to maintain per-person identity and motion consistency across frequent video shot changes significantly improves 3D human mesh recovery and camera pose estimation. You should consider integrating this approach to enhance the fidelity and temporal coherence of your tracking results in complex, real-world footage.

Key insights

Multi-THuMBS tracks multi-person 3D human meshes across video shot changes using a shared 3D scene prior for consistent identity and motion.

Principles

• Complex interactions and occlusions challenge 3D human mesh tracking.
• Shot changes cause identity loss and inconsistent tracking trajectories.
• Single-person tracking fails in multi-person real-world video scenarios.

Method

Multi-THuMBS reconstructs two boundary frames using a 3D scene prior in a shared 3D space. Human meshes are then registered within this space, preserving per-person identity and motion consistency across shot changes.

Topics

• Multi-person Tracking
• 3D Human Meshes
• Video Shot Transitions
• 3D Scene Reconstruction
• Camera Pose Estimation
• Identity Preservation

Best for: Research Scientist, AI Scientist, Computer Vision Engineer

Related on AIssential

• Multi-HMR 2: Multi-Person Camera-Centric Human Detection, Mesh Recovery and Tracking — Multi-HMR 2 integrates camera-centric detection, mesh recovery, and tracking for robust 3D human perception.
• Mesh-Aware Epipolar Matching for Multi-View Multi-Person 3D Pose Estimation in Basketball — Dense mesh geometry significantly enhances training-free multi-view 3D pose estimation in complex sports environments.
• MVTrack4Gen: Multi-View Point Tracking as Geometric Supervision for 4D Video Generation — MVTrack4Gen uses multi-view point tracking to provide geometric and motion supervision for novel-view video diffusion models.
• DisPOSE: Projected Polystochastic Diffusion for Self-Supervised Multi-View 3D Human Pose Estimation — Self-supervised multi-view 3D human pose estimation can be achieved by modeling discrete person assignment as projected diffusion on polystochastic tensors.
• SAM 3.1: Faster and More Accessible Real-Time Video Detection and Tracking With Multiplexing and Global Reasoning - AI at Meta — SAM 3.1 enhances real-time video object tracking via multiplexing, doubling processing speed and reducing GPU demands.
• TextHOI-3D: Text-to-3D Hand-Object Interaction via Discrete Multi-View Generation and Joint Mesh Optimization — Multi-view visual tokens effectively bridge text-conditioned visual generation and geometry-aware 3D hand-object mesh recovery.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.