Superman: Unifying Skeleton and Vision for Human Motion Perception and Generation

2026-07-08 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

Superman is a novel unified framework designed to bridge the gap between human motion perception and generation tasks in computer vision. It addresses the current fragmentation where models either understand motion from video (outputting text) or generate motion from non-visual inputs, often limited to static poses. The framework introduces a Vision-Guided Motion Tokenizer, which utilizes geometric alignment between 3D skeletons and visual data to create a unified, cross-modal motion vocabulary. Grounded in this vocabulary, a single Multi-modal Large Language Model (MLLM) architecture, based on Qwen2.5-VL-7B, is trained to perform 3D pose estimation from video, motion prediction, and motion in-betweening. Extensive experiments on Human3.6M and 3DPW benchmarks demonstrate that Superman achieves leading or competitive performance across all tasks, including an 11.97% improvement for 3D pose estimation on Human3.6M over prior multi-task perception methods.

Key takeaway

For Computer Vision Engineers developing human motion analysis systems, this unified framework offers a compelling alternative to fragmented perception and generation models. You should consider adopting a visually-grounded, multi-task MLLM approach to streamline your workflows for 3D pose estimation, motion prediction, and in-betweening. This can lead to more efficient development and superior performance, as demonstrated by Superman's strong results on standard benchmarks.

Key insights

Superman unifies human motion perception and generation by creating a visually-grounded, cross-modal motion language for a single MLLM.

Principles

• Unifying perception and generation improves performance.
• Cross-modal tokenization enhances motion representation.
• Larger models and codebooks yield better accuracy.

Method

A Vision-Guided Motion Tokenizer (VQ-VAE) fuses visual and skeletal data into a hybrid codebook. An MLLM (Qwen2.5-VL-7B) then autoregressively generates motion tokens for perception and generation tasks.

In practice

• Estimate 3D pose from video.
• Predict future motion sequences.
• In-between missing motion frames.

Topics

• Human Motion Analysis
• 3D Pose Estimation
• Motion Prediction
• Motion In-betweening
• Multi-modal LLMs
• Vision-Guided Tokenizer

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Ego-Human Motion Prediction with 3D-Aware LLM — Accurate egocentric motion prediction requires holistic 3D spatial-semantic understanding and integrated pose-language decoding.
• TROPHIES: Temporal Reconstruction of Places, Humans, and Cameras from Multi-view Videos — TROPHIES unifies human, scene, and camera reconstruction from multi-view videos into a globally consistent 4D space.
• Universal Skeleton Understanding via Differentiable Rendering and MLLMs — SkeletonLLM enables MLLMs to understand non-visual skeleton data by converting it into visual sequences via differentiable rendering.
• Beyond Skeletons: Learning Animation Directly from Driving Videos with Same2X Training Strategy — DirectAnimator bypasses pose estimation in HIA by using raw video cues and a specialized training strategy for superior results.
• 🐪DuoMo: Dual Motion Diffusion🐪 👉DuoMo by META is a novel generative method that recovers... — DuoMo recovers human motion in world-space from noisy, unconstrained video using a novel dual motion diffusion approach.
• COMPOSE: Hypergraph Cover Optimization for Multi-view 3D Human Pose Estimation — Hypergraph partitioning for multi-view 3D pose estimation improves accuracy and robustness over pairwise methods.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.