MilliVid: Hierarchical Latents for Long-Range Consistency in Video Generation

2026-06-08 · Source: Machine Learning · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision & Pattern Recognition · Depth: Expert, quick

Summary

MilliVid is a novel video generation model introduced to overcome long-range consistency challenges in generative video models, which typically struggle with impractically long transformer sequence lengths for extended videos. This approach employs a coarse-to-fine rollout strategy within a multi-scale token space. First, an autoencoder is pre-trained to compress each video frame into a hierarchical token structure, ranging from standard latent resolutions to just a few tokens per frame. The coarsest tokens encode critical information like scene layout and semantics, while finer tokens capture high-frequency appearance and texture. Subsequently, a video diffusion model is trained to generate these tokens using the coarse-to-fine rollout. This method precisely manages the level of detail generated and used as context, ensuring long-range consistency in geometry and object permanence, while optimizing compute by focusing on perceptually relevant details. MilliVid was validated using a custom dataset of long Minecraft videos, demonstrating significantly more consistent rollouts compared to existing baselines.

Key takeaway

For Machine Learning Engineers developing video generative models, if you are struggling with long-range consistency or high computational demands for extended sequences, consider adopting a hierarchical latent approach. MilliVid's method of compressing frames into multi-scale tokens and using coarse-to-fine rollout can significantly improve object permanence and geometric consistency. You should explore implementing similar multi-scale tokenization to optimize compute while maintaining visual coherence in your long-form video outputs.

Key insights

Hierarchical latent token compression and coarse-to-fine rollout enable long-range consistency in video generation by managing detail levels efficiently.

Principles

• Coarse tokens encode scene layout and semantics.
• Finer tokens add high-frequency appearance and texture.
• Detail-level control preserves long-range consistency.

Method

Pre-train an autoencoder to compress frames into hierarchical tokens. Then, train a video diffusion model to generate these tokens via coarse-to-fine rollout, controlling detail and context for consistency.

In practice

• Generate long, consistent videos for virtual environments.
• Improve object permanence in extended generative sequences.
• Reduce computational load for high-fidelity video.

Topics

• Video Generation
• Long-Range Consistency
• Hierarchical Latents
• Diffusion Models
• Autoencoders
• MilliVid

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Consistency-Preserving Diverse Video Generation — A new flow-matching framework improves video diversity and temporal consistency using latent-space gradient regulation.
• (1D) Ordered Tokens Enable Efficient Test-Time Search — 1D ordered tokens with coarse-to-fine structure significantly enhance test-time search in autoregressive image generation.
• EntityBench: Towards Entity-Consistent Long-Range Multi-Shot Video Generation — Maintaining entity consistency across long multi-shot video generation remains a significant challenge for current methods.
• [R] Dynamic Large Concept Models: Latent Reasoning in an Adaptive Semantic Space — Dynamic Large Concept Models explore latent generative modeling for text, a technique common in image/video diffusion.
• End-to-End Training for Unified Tokenization and Latent Denoising — Unified tokenization and latent diffusion can be jointly optimized in a single-stage training process.
• Stream3D: Sequential Multi-View 3D Generation via Evidential Memory — Stream3D enables consistent 3D generation from long video streams by selectively caching evidential views, avoiding latent state transport.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning.