VideoFlexTok: Flexible-Length Coarse-to-Fine Video Tokenization

2026-07-02 · Source: Apple Machine Learning Research · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision · Depth: Expert, quick

Summary

VideoFlexTok, a novel video tokenization method published in July 2026 by researchers including Andrei Atanov and Amir Zamir, introduces a variable-length, coarse-to-fine token sequence representation for videos. Unlike standard 3D grid tokenization that demands "pixel-by-pixel" learning, VideoFlexTok's initial tokens capture abstract information like semantics and motion, with subsequent tokens adding fine-grained details. Its generative flow decoder facilitates realistic video reconstructions from any token count, allowing adaptation to downstream needs and efficient encoding of longer videos. Evaluations on class- and text-to-video generative tasks demonstrate more efficient training, achieving comparable generation quality (gFVD and ViCLIP Score) with a 5x smaller model (1.1B vs 5.2B). This approach also enables long video generation, exemplified by training a text-to-video model on 10-second, 81-frame videos using only 672 tokens, an 8x reduction compared to 3D grid tokenizers.

Key takeaway

For Machine Learning Engineers developing video generative models, VideoFlexTok offers a path to significantly improve training efficiency and scalability. You can achieve comparable generation quality with 5x smaller models, reducing computational costs. Consider adopting coarse-to-fine, variable-length tokenization to enable longer video generation without prohibitive resource demands. This approach allows you to adapt token counts precisely to your specific application's detail requirements.

Key insights

VideoFlexTok uses variable-length, coarse-to-fine tokens for efficient video representation and generation, reducing model complexity.

Principles

• Video tokenization can capture abstract to fine-grained details.
• Decoupling video duration from token count improves efficiency.
• Variable-length token sequences adapt to diverse downstream needs.

Method

VideoFlexTok represents videos with a variable-length, coarse-to-fine token sequence. A generative flow decoder reconstructs videos from any token count, allowing adaptive detail levels and efficient long video encoding.

In practice

• Generate 10-second, 81-frame videos with 672 tokens.
• Achieve comparable quality with 5x smaller generative models.
• Adapt token count based on specific downstream task requirements.

Topics

• VideoFlexTok
• Video Tokenization
• Coarse-to-Fine Representation
• Generative Video Models
• Model Efficiency
• Long Video Generation

Best for: Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• MilliVid: Hierarchical Latents for Long-Range Consistency in Video Generation — Hierarchical latent token compression and coarse-to-fine rollout enable long-range consistency in video generation by managing detail levels efficiently.
• Variable-Length Tokenization via Learnable Global Merging for Diffusion Transformers — Merging similar tokens enables variable-length tokenization while preserving cross-length representation alignment in Diffusion Transformers.
• EVATok: Adaptive Length Video Tokenization for Efficient Visual Autoregressive Generation — EVATok adaptively tokenizes video segments to optimize quality-cost trade-offs in autoregressive video generation.
• FlexAM: Flexible Appearance-Motion Decomposition for Versatile Video Generation Control — Disentangling appearance and motion via a 3D point cloud control signal improves video generation control.
• (1D) Ordered Tokens Enable Efficient Test-Time Search — 1D ordered tokens with coarse-to-fine structure significantly enhance test-time search in autoregressive image generation.
• InsightTok: Improving Text and Face Fidelity in Discrete Tokenization for Autoregressive Image Generation — Localized, content-aware perceptual losses significantly enhance text and face fidelity in discrete image tokenization.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Apple Machine Learning Research.