LiveStre4m: Feed-Forward Live Streaming of Novel Views from Unposed Multi-View Video

2026-04-08 · Source: Takara TLDR - Daily AI Papers · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Robotics & Autonomous Systems · Depth: Expert, medium

Summary

LiveStre4m is a novel feed-forward method for real-time live streaming of novel view synthesis (NVS) from unposed, sparse multi-view video inputs. Developed by Pedro Quesado and colleagues, this system addresses the limitations of existing dynamic scene representation methods that require ground-truth camera parameters and lengthy optimizations, making them unsuitable for live applications. LiveStre4m integrates a multi-view vision transformer for keyframe 3D scene reconstruction and a diffusion-transformer interpolation module to ensure temporal consistency. Crucially, it includes a Camera Pose Predictor that estimates camera poses and intrinsics directly from RGB images, eliminating the need for prior calibration. The system achieves an average reconstruction time of 0.07 seconds per frame at 1024x768 resolution, significantly outperforming optimization-based methods and enabling real-time NVS streaming with as few as two synchronized unposed input streams.

Key takeaway

For research scientists developing real-time 3D reconstruction or live streaming applications, LiveStre4m offers a significant advancement by enabling novel view synthesis from unposed multi-view video without extensive optimization or prior camera calibration. You should explore its feed-forward architecture and Camera Pose Predictor to overcome latency and setup complexities in your projects, potentially integrating its principles for more deployable real-time systems.

Key insights

LiveStre4m enables real-time novel view synthesis from unposed multi-view video using a feed-forward, calibration-free approach.

Principles

• Feed-forward models enable real-time NVS.
• Camera pose prediction removes calibration dependency.
• Diffusion-transformers ensure temporal consistency.

Method

LiveStre4m uses a multi-view vision transformer for 3D reconstruction, a diffusion-transformer for temporal consistency, and a Camera Pose Predictor to estimate poses and intrinsics directly from RGB images, enabling real-time NVS.

In practice

• Stream novel views from two unposed cameras.
• Achieve 0.07s/frame NVS at 1024x768.
• Integrate uncalibrated camera feeds.

Topics

• Novel View Synthesis
• Live Streaming
• Multi-View Video
• Camera Pose Estimation
• Vision Transformers

Code references

• pedro-quesado/LiveStre4m
• mit-han-lab/streaming-vlm

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• MoRe: Motion-aware Feed-forward 4D Reconstruction Transformer — MoRe efficiently reconstructs dynamic 4D scenes from monocular video by disentangling motion from static structure.
• LiveStarPro: Proactive Streaming Video Understanding with Hierarchical Memory for Long-Horizon Streams — LiveStarPro enables proactive, real-time video understanding for long streams by integrating hierarchical memory and efficient decoding.
• Novel View Synthesis as Video Completion — Video diffusion models can be adapted for sparse novel view synthesis by treating it as low frame-rate video completion.
• LongStream: Long-Sequence Streaming Autoregressive Visual Geometry — LongStream enables stable, metric-scale 3D reconstruction for ultra-long sequences by decoupling gauge and optimizing Transformer caches.
• Feed-Forward 3D Scene Modeling: A Problem-Driven Perspective — Feed-forward 3D reconstruction offers efficient, generalizable scene modeling by directly mapping 2D inputs to 3D representations.
• 🪝Syn4D: Multiview Synthetic 4D Dataset🪝 👉Syn4D is novel multi-view synthetic dataset of... — Syn4D is a synthetic multi-view 4D dataset offering comprehensive ground-truth for dynamic scene analysis.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Takara TLDR - Daily AI Papers.