ProxyPose: 6-DoF Pose Tracking via Video-to-Video Translation

2026-07-08 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

ProxyPose introduces a novel approach to six-degree-of-freedom (6-DoF) pose tracking, recasting it as a video-to-video translation problem using a fine-tuned video diffusion model. Unlike existing methods that require 3D models, depth maps, or object masks, ProxyPose operates solely from a monocular video and a single marked pixel in the first frame. The system translates the input into a "proxy video" depicting a colored polyhedron undergoing the same local rigid-body motion. Its known geometry allows classical pose estimation solvers to recover the full 6-DoF trajectory. This method leverages large-scale video pre-training to handle challenging materials, occlusions, and deformations at the pixel level, achieving state-of-the-art accuracy on HO3D and YCBInEOAT datasets, and extending to face tracking and camera pose estimation in diverse, in-the-wild scenes.

Key takeaway

For Computer Vision Engineers developing robust 6-DoF tracking solutions, you should consider integrating video-to-video translation with diffusion models. This approach significantly reduces input requirements, needing only monocular RGB video and a single marked pixel, and demonstrates superior accuracy and robustness on challenging surfaces and occlusions compared to methods relying on explicit 3D models or depth sensors. Explore fine-tuning pre-trained video models for similar perception tasks.

Key insights

Video diffusion models can implicitly encode 3D motion for 6-DoF pose tracking via video-to-video translation.

Principles

• Large video models implicitly encode 3D motion information.
• Recasting hard perception problems as translation simplifies 6-DoF recovery.

Method

Fine-tune a video diffusion model to translate input video into a proxy video (colored polyhedron). Recover 6-DoF pose from the proxy using classical PnP solvers and temporal smoothness optimization.

In practice

• Track 6-DoF pose for textureless, transparent, or deformable surfaces.
• Estimate camera pose from static background regions.
• Perform face tracking without a face-specific model.

Topics

• 6-DoF Pose Tracking
• Video-to-Video Translation
• Video Diffusion Models
• Monocular Vision
• Computer Vision
• Generative Models

Code references

• modelscope/DiffSynth-Studio

Best for: Research Scientist, AI Scientist, Computer Vision Engineer, Robotics Engineer

Related on AIssential

• Video Reconstruction using Diffusion-based Image-to-Video Generation with Trajectory Guidance — Trajectory-guided diffusion models can reconstruct missing video frames by integrating external sensor data.
• One-Shot Novel View and Pose Human Image Synthesis via 3D Prior Guided Diffusion Model — A 3D prior-guided conditional diffusion model enables high-quality one-shot novel view and pose human image synthesis.
• MVTrack4Gen: Multi-View Point Tracking as Geometric Supervision for 4D Video Generation — MVTrack4Gen uses multi-view point tracking to provide geometric and motion supervision for novel-view video diffusion models.
• Masked Diffusion Vision-Language Models for Temporal Action Localization — Masked diffusion vision-language models enable joint, iterative refinement of temporal boundaries and semantic content in untrimmed videos.
• DPPE: Rethinking Camera-Based Positional Encoding for Scaling Multi-View Transformers — Decoupling rotation and translation in camera-based positional encoding prevents training stagnation in scaled multi-view Transformers for NVS.
• Latent Spatial Memory for Video World Models — Latent spatial memory in diffusion models significantly boosts video generation speed and reduces memory by avoiding pixel-space reconstruction.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.