Video = World + Event Stream

2026-07-16 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, quick

Summary

Wan-Streamer v0.3 introduces a novel interaction model, reframing video as a "world plus an event stream." The "world" represents the persistent context, encompassing elements like the environment, scene, subjects, and ambient acoustic conditions. The "event stream" captures all dynamic changes over time, including scene shifts, subject behavior, and speech. This framework enables a general-purpose pretraining task: predicting real-time world movements, changes, and responses given a world and incoming input. The model specializes in real-time full-duplex audio-visual interaction, mapping multimodal user input to language-form speech and behavior actions. Wan-Streamer v0.3 maintains its v0.2 operating point, supporting 640x368 video at 25 FPS, with a 160 ms streaming unit, approximately 200 ms model-side response latency, and about 550 ms total interaction latency under a 350 ms bidirectional network budget.

Key takeaway

For Machine Learning Engineers developing real-time interactive AI, Wan-Streamer v0.3's "world + event stream" paradigm offers a robust framework for predictive modeling. You should consider this approach for systems requiring low-latency audio-visual interaction, as it achieves approximately 200 ms model-side response latency. This design can streamline your development of agents that map multimodal input to dynamic speech and behavior actions.

Key insights

Wan-Streamer v0.3 models video as a persistent "world" and dynamic "event stream" for real-time predictive interaction.

Principles

• Video understanding benefits from separating static context from dynamic changes.
• Predicting real-time world responses enables broad downstream tasks.
• Multimodal input can map to language-form speech and behavior actions.

Method

The model's pretraining task involves predicting real-time world movements and responses given a persistent world context and incoming event stream input.

In practice

• Apply to real-time full-duplex audio-visual interaction.
• Utilize for agent speech and free-form behavior generation.
• Achieve 200 ms model latency for interactive systems.

Topics

• Wan-Streamer v0.3
• Video Understanding
• Real-time Interaction
• Multimodal AI
• Event Stream Modeling

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Wan-Streamer v0.2: Higher Resolution, Same Latency — Wan-Streamer v0.2 achieves higher resolution interactive video (640x368) with preserved low latency (200ms model-side) via a split serving architecture.
• Proact-VL: A Proactive VideoLLM for Real-Time AI Companions — Proact-VL enables human-like AI companions for real-time video by balancing proactive timing with controlled, low-latency content generation.
• Interaction Models: A Scalable Approach to Human-AI Collaboration — Native, real-time multimodal interaction within AI models enhances human-AI collaboration beyond turn-based systems.
• Modularized Dynamic-Granularity Video LLM for Multi-Event Long Video Understanding — MoD-VLLM iteratively grounds and understands long videos using dynamic granularity and self-reflection to overcome token budget limits.
• [AINews] Thinking Machines' Native Interaction Models - TML-Interaction-Small 276B-A12B - advances SOTA Realtime Voice and kills standard VAD — Real-time, full-duplex multimodal interaction models are advancing human-AI collaboration beyond turn-based systems.
• Thinking Machines shows off preview of near-realtime AI voice and video conversation with new 'interaction models' — Thinking Machines' interaction models enable real-time, full-duplex human-AI communication by processing multimodal inputs and outputs simultaneously.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.