HandsOnWorld: Unconstrained Egocentric Video Generation with Camera-Disentangled Hand Control

2026-07-02 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision · Depth: Expert, quick

Summary

HandsOnWorld is a novel framework designed for hand-controlled egocentric video generation, overcoming limitations of multi-view and marker-based motion capture by learning from unconstrained monocular video. This approach addresses the scarcity of scalable 3D hand annotations, which previously confined hand-controlled generators to restricted scene distributions. The framework introduces a protagonist-centered annotation pipeline that filters monocular reconstructions at action-semantic, image-quality, and 3D-geometric levels. This process creates EgoVid-Pro, a dataset comprising 103K clips and approximately 12M frames of clean, protagonist-only hand trajectories across diverse everyday scenes. To resolve camera-hand entanglement caused by large ego-motion, HandsOnWorld proposes the Plücker Hand Map, a 3D-aware control signal that extends Plücker-ray representations to the hand surface, effectively disentangling camera and hand motion at the representation level. Experiments demonstrate superior reconstruction fidelity and control accuracy compared to prior methods, with strong generalization to out-of-distribution environments.

Key takeaway

For Computer Vision Engineers developing egocentric video generation systems, HandsOnWorld offers a robust solution for unconstrained hand control. You should consider integrating its protagonist-centered annotation pipeline to create high-quality 3D hand datasets from diverse monocular video. Furthermore, adopting the Plücker Hand Map can effectively resolve camera-hand entanglement, leading to more accurate and generalizable hand-controlled video synthesis beyond laboratory settings. This approach significantly expands possibilities for realistic human-computer interaction simulations.

Key insights

HandsOnWorld disentangles camera and hand motion for egocentric video generation via a novel 3D-aware control signal and a large, clean dataset.

Principles

• Unconstrained monocular video enables hand-controlled generation.
• Protagonist-centered filtering improves 3D hand annotation quality.
• Plücker Hand Map disentangles camera and hand motion.

Method

HandsOnWorld reconstructs 3D hands from monocular video using a protagonist-centered annotation pipeline, filtering reconstructions for quality. It then employs the Plücker Hand Map to disentangle camera and hand motion for egocentric video generation.

In practice

• Generate egocentric videos with precise, unconstrained hand control.
• Build large 3D hand datasets from diverse monocular footage.
• Apply Plücker Hand Maps for camera-hand motion disentanglement.

Topics

• Egocentric Video Generation
• Hand Tracking
• Monocular 3D Reconstruction
• Plücker Hand Map
• EgoVid-Pro Dataset
• Computer Vision

Best for: Research Scientist, AI Scientist, Computer Vision Engineer

Related on AIssential

• WHOLE: World-Grounded Hand-Object Lifted from Egocentric Videos — Jointly modeling hand-object interactions with a generative prior improves egocentric video reconstruction.
• StableHand: Quality-Aware Flow Matching for World-Space Dual-Hand Motion Estimation from Egocentric Video — Accurate dual-hand motion estimation from egocentric video requires explicit per-frame observation quality awareness.
• Ego2World: Compiling Egocentric Cooking Videos into Executable Worlds for Belief-State Planning — Ego2World enables embodied agents to plan and adapt in partially observed, dynamic environments by compiling real egocentric videos into executable symbolic worlds.
• UMI-3D: Extending Universal Manipulation Interface from Vision-Limited to 3D Spatial Perception — UMI-3D enhances robot manipulation data collection via LiDAR-centric SLAM and multimodal sensor fusion.
• Hand Trajectory Fusion for Egocentric Natural Language Query Grounding — Integrating hand trajectory data significantly improves egocentric video NLQ grounding for interaction-focused queries.
• EgoPressDiff: Multimodal Video Diffusion for Egocentric UV-Domain Hand-Pressure Estimation — EgoPressDiff uses multimodal video diffusion to generate continuous, temporally consistent UV-pressure maps from egocentric video.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.