StreamHOI: Interaction-aware Temporal Memory Adaptation for Streaming HOI Video Generation

2026-07-22 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision · Depth: Expert, quick

Summary

StreamHOI is a novel low-latency streaming framework designed for long-duration human-object interaction (HOI) video generation, addressing the limitations of existing offline, short-video methods. Unlike converting complex HOI pipelines, StreamHOI focuses on optimizing historical memory organization within an image-to-video streaming generator to preserve interactions under bounded latency. The research identifies that standard sink-local memory designs present trade-offs, and different transformer blocks exhibit distinct historical-memory preferences for HOI and surrounding regions. To counter this, StreamHOI employs offline HOI-aware block profiling and bias-guided memory-specialized training, adapting the generator to block-specific memory layouts. It also integrates a memory distance scaling module to enhance long-range access to early interaction states. StreamHOI demonstrates strong interaction plausibility, object fidelity, human quality, and efficiency, achieving 17.6 FPS with a 0.75-second first-chunk latency.

Key takeaway

For Machine Learning Engineers developing real-time interactive human-object interaction (HOI) applications, StreamHOI presents a significant advancement. You should evaluate its approach to temporal memory adaptation, which enables low-latency, long-duration video generation. By profiling transformer block memory preferences and applying bias-guided training, you can achieve strong interaction plausibility and efficiency, reaching 17.6 FPS with minimal first-chunk latency. This framework offers a robust solution for streaming HOI video.

Key insights

StreamHOI optimizes streaming HOI video generation by adapting temporal memory to transformer block preferences for low-latency, long-duration output.

Principles

• Standard sink-local memory has trade-offs in streaming HOI.
• Transformer blocks show varied memory preferences for HOI regions.
• Adapting memory composition to block behavior improves HOI generation.

Method

StreamHOI performs offline HOI-aware block profiling, applies bias-guided memory-specialized training, and introduces a memory distance scaling module to strengthen long-range access.

In practice

• Profile transformer blocks for HOI-specific memory needs.
• Train generators with bias-guided, block-specific memory layouts.
• Implement memory distance scaling for long-range interaction states.

Topics

• Human-Object Interaction
• Video Generation
• Streaming AI
• Temporal Memory Adaptation
• Transformer Networks
• Low-Latency Systems

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• StreamHOI: Interaction-aware Temporal Memory Adaptation for Streaming HOI Video Generation — StreamHOI adapts Transformer block memory based on spatial attention bias for consistent, low-latency, long-duration HOI video generation.
• SparseCtrl-HOI: Sparse Temporal Control for Human-Object Interaction Video Generation — Sparse temporal control with MLLM-driven motion priors enables realistic HOI video generation with reduced annotation.
• Video = World + Event Stream — Wan-Streamer v0.3 models video as a persistent "world" and dynamic "event stream" for real-time predictive interaction.
• Taming I2V models for Image HOI Editing: A Cognitive Benchmark and Agentic Self-Correcting Framework — A new benchmark and agentic self-correcting framework using I2V models significantly improve complex Human-Object Interaction (HOI) image editing.
• Stream3D: Sequential Multi-View 3D Generation via Evidential Memory — Stream3D uses an evidential memory and evidence score mechanism to enable consistent 3D generation from sequential multi-view inputs without retraining existing models.
• LiveStarPro: Proactive Streaming Video Understanding with Hierarchical Memory for Long-Horizon Streams — LiveStarPro enables proactive, real-time video understanding for long streams by integrating hierarchical memory and efficient decoding.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.