Group-of-Latents: Perceptual Video Compression at Extreme Bitrates via Masked Latent Generative Modeling

2026-03-30 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Expert, extended

Summary

Peking University researchers introduce Group-of-Latents (GoL), a unified generative framework for extreme-low bitrate video compression, operating below 0.005 bpp. This method leverages pre-trained Diffusion Transformer (DiT) priors and a causal tokenizer to explicitly partition the latent stream into intra I-latents and inter P-latents. The Deep Compression Module (I-DCM) encodes key I-latents for perceptual anchors with minimal overhead. Subsequently, the DiT-based Unified Latent Denoising Module (U-LDM) refines intra-frame textures and synthesizes P-latents from noise, reconstructing temporal dynamics at zero additional bitrate. Experiments demonstrate state-of-the-art perceptual fidelity, rich spatial details, and robust temporal consistency, outperforming traditional and neural codecs in the extreme-low bitrate regime. The framework supports dynamic bitrate adjustment without retraining.

Key takeaway

For Machine Learning Engineers developing video streaming or storage solutions, this GoL framework offers a critical advancement for extreme-low bitrate scenarios. You should consider integrating this approach to achieve superior perceptual quality and temporal consistency below 0.005 bpp, where traditional codecs fail. Its dynamic GoL configuration allows flexible bitrate adaptation without model retraining, simplifying deployment for varying bandwidth conditions.

Key insights

Decoupling spatial and temporal video latents with generative priors enables extreme-low bitrate perceptual compression.

Principles

• Explicitly decouple spatial texture and temporal motion in latent space.
• Leverage generative priors for perceptual quality at extreme bitrates.
• Dynamic GoL allows flexible bitrate adaptation without retraining.

Method

Partition latent stream into I-latents (compressed by I-DCM) and P-latents (synthesized from noise by U-LDM using DiT priors), guided by text.

In practice

• Use GoL masks to dynamically adjust video compression bitrates.
• Apply DiT-based denoising for perceptual refinement of compressed latents.

Topics

• Video Compression
• Perceptual Compression
• Diffusion Models
• Diffusion Transformer
• Group-of-Latents
• Extreme Bitrates

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• Group-of-Latents: Perceptual Video Compression at Extreme Bitrates via Masked Latent Generative Modeling — GoL uses DiT priors for extreme-low-bitrate perceptual video compression, partitioning latents for efficient reconstruction.
• Exploiting Semantic and Pixel Representations for Ultra-Low Bitrate Image Compression — Ultra-low bitrate image compression can achieve pixel-level fidelity by combining semantic and pixel representations in a diffusion model.
• Closing the Loop: Training-Free Revisit Consistency for Autoregressive Generative Rendering — Training-free geometric and temporal correspondence improves autoregressive generative rendering consistency during camera revisits.
• Blind Quality Enhancement of Compressed Video via Fine-Grained Degradation-Guided Sequential Inference — Fine-grained, multi-scale degradation representation and adaptive processing stages significantly improve blind compressed video quality enhancement.
• Frozen Forecasting: A Unified Evaluation — Perceptual ability in frozen video models strongly correlates with their generalist forecasting performance over short time horizons.
• Novel View Synthesis as Video Completion — Video diffusion models can be adapted for sparse novel view synthesis by treating it as low frame-rate video completion.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.