SparseCtrl-HOI: Sparse Temporal Control for Human-Object Interaction Video Generation

2026-07-08 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, extended

Summary

SparseCtrl-HOI is a new sparse temporal control framework designed for Human-Object Interaction (HOI) video generation, addressing the high annotation costs and rigid motions of prior dense guidance methods. Developed by South China University of Technology, this framework requires only a few keyframes to define interaction states. It integrates a Time-Controlled Rotary Positional Embedding (TiRoPE) for precise temporal anchoring and a Motion Prior Injection Module, which uses Multimodal Large Language Models (MLLMs) like Qwen2.5-VL and a Q-Former to extract high-level motion priors for natural intermediate transitions. A decoupled two-stage training strategy further optimizes its performance. The researchers also introduced SparseHOI-5K, a public dataset of 4,850 clips with rich annotations. Evaluations show SparseCtrl-HOI significantly reduces annotation overhead and produces superior, physically plausible HOI videos for live-streaming e-commerce, outperforming state-of-the-art baselines on FID, FVD, MS-RAFT, and HOI-VLM metrics. Code and dataset are publicly available.

Key takeaway

For AI Engineers developing human-object interaction video generation systems, you should consider SparseCtrl-HOI's sparse temporal control paradigm. This approach significantly reduces annotation overhead by requiring only keyframes, while its MLLM-driven motion priors ensure natural, physically plausible transitions. Adopting this framework can enhance the realism and scalability of your AI-driven avatars for applications like live-streaming e-commerce, overcoming the rigid motions often seen with dense guidance.

Key insights

Sparse temporal control with MLLM-driven motion priors enables realistic HOI video generation with reduced annotation.

Principles

• Dense temporal guidance limits motion diversity.
• Decoupled training prevents feature entanglement.
• MLLMs can infer high-level motion semantics.

Method

SparseCtrl-HOI anchors keyframes with TiRoPE and injects MLLM-derived motion priors via a Q-Former into a DiT backbone, using a two-stage decoupled training strategy.

In practice

• Define interaction states with sparse keyframes.
• Utilize MLLMs for high-level motion prior extraction.
• Implement decoupled training for complex generative models.

Topics

• Human-Object Interaction
• Video Generation
• Sparse Temporal Control
• MLLM Motion Priors
• Diffusion Transformers
• SparseHOI-5K Dataset

Best for: Research Scientist, Computer Vision Engineer, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• HOI-aware Adaptive Network for Weakly-supervised Action Segmentation — Leveraging human-object interaction (HOI) context dynamically improves weakly-supervised action segmentation accuracy for ambiguous actions.
• Modality Forcing for Scalable Spatial Generation — Modality Forcing enables scalable, joint image-depth generation using sparse data and T2I models, outperforming prior methods.
• Masked Diffusion Vision-Language Models for Temporal Action Localization — Masked diffusion vision-language models enable joint, iterative refinement of temporal boundaries and semantic content in untrimmed videos.
• Policy-as-Data: Learning Generalizable HOI Diffusion Models from Simulated Physics — Leveraging physics simulation and RL for data generation improves HOI model generalization and physical consistency.
• Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understanding — Video generation models inherently learn robust 3D structural priors and physical laws for scene understanding.
• Moment-Video: Diagnosing Temporal Fidelity of Video MLLMs on Momentary Visual Events — Current video MLLMs struggle with momentary visual events due to sampling, compression, and aggregation issues, lacking temporal fidelity.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.