StreamHOI: Interaction-aware Temporal Memory Adaptation for Streaming HOI Video Generation

2026-07-23 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems, Emerging Technologies & Innovation · Depth: Expert, extended

Summary

StreamHOI is a novel low-latency streaming framework designed for long-duration human-object interaction (HOI) video generation, addressing limitations of existing offline short-video methods. It identifies that different Transformer blocks exhibit distinct historical-memory preferences for HOI regions versus surrounding areas. StreamHOI leverages this by performing offline HOI-aware block profiling to categorize blocks. Subsequently, bias-guided memory-specialized training (B-MST) adapts the generator to block-specific sink-local memory layouts. It allocates more sink memory to HOI-biased blocks for consistency and more local memory to surrounding-biased blocks for continuity. A memory distance scaling (MDS) module further strengthens long-range access to early interaction states. StreamHOI achieves 17.6 FPS with 0.75s first-chunk latency, demonstrating strong interaction plausibility, object fidelity, and human quality. It outperforms long-video baselines and competes effectively with bidirectional HOI methods. It maintains consistency over 20,000 frames and supports dynamic prompt switching.

Key takeaway

For AI Scientists and Machine Learning Engineers building real-time, long-duration human-object interaction video generation systems, move beyond uniform memory management. This research demonstrates that profiling Transformer block attention to identify HOI-biased and surrounding-biased behaviors significantly enhances interaction plausibility and object fidelity. Apply specialized memory layouts and distance scaling. Implement block-specific memory adaptation to achieve robust, scalable HOI generation without increasing your total KV-cache budget.

Key insights

StreamHOI adapts Transformer block memory based on spatial attention bias for consistent, low-latency, long-duration HOI video generation.

Principles

• Transformer blocks have distinct HOI-region vs. surrounding-region memory preferences.
• Memory composition should align with block-wise spatial bias for optimal HOI generation.
• Long-range memory access is crucial for preserving early interaction states.

Method

Profile Transformer blocks for HOI-biased or surrounding-biased attention. Train with block-specific sink-local memory layouts and memory distance scaling for HOI-biased blocks.

In practice

• Profile attention patterns to identify block-specific memory needs.
• Allocate more sink memory to HOI-focused blocks for consistency.
• Use memory distance scaling to improve access to distant interaction states.

Topics

• Human-Object Interaction
• Streaming Video Generation
• Diffusion Transformers
• Temporal Memory Adaptation
• Block-wise Memory Organization
• Low-Latency Inference
• Video Consistency

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• StreamHOI: Interaction-aware Temporal Memory Adaptation for Streaming HOI Video Generation — StreamHOI optimizes streaming HOI video generation by adapting temporal memory to transformer block preferences for low-latency, long-duration output.
• AgentHOI: Multi-Agent Reasoning for Human-Object-Interaction Video Generation via Implicit Representation Alignment — Text-driven HOI video generation benefits from multi-agent reasoning and implicit text-motion alignment, eliminating explicit motion inputs.
• HeadCast: Casting Attention Heads for Efficient Autoregressive Video Generation — Attention heads in AR video diffusion models exhibit stable, heterogeneous behaviors that can be exploited for efficient inference.
• Stream3D: Sequential Multi-View 3D Generation via Evidential Memory — Stream3D uses an evidential memory and evidence score mechanism to enable consistent 3D generation from sequential multi-view inputs without retraining existing models.
• Microsoft Research's Mirage gives video generation a persistent spatial memory that doesn't forget what's around the corner — Mirage uses latent spatial memory to achieve persistent spatial consistency and efficiency in video world models.
• A Study of Failure Modes in Two-Stage Human-Object Interaction Detection — HOI models struggle with multi-person scenes and object-conditioned verb biases, leading to persistent, high-confidence errors.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.