Accelerating Text-to-Video Generation with Calibrated Sparse Attention

2026-07-21 · Source: Apple Machine Learning Research · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

CalibAtt is a novel, training-free method designed to accelerate text-to-video generation by optimizing spatiotemporal attention in diffusion models. These models, while producing high-quality video, are bottlenecked by slow runtimes due to large transformer-based backbones. The method identifies that a significant fraction of token-to-token connections consistently yield negligible scores and exhibit repeating patterns across queries and local token blocks, allowing their attention computation to be skipped. CalibAtt performs an offline calibration pass to detect these stable block-level sparsity and repetition patterns across inputs, compiling them into optimized attention operations for each layer, head, and diffusion timestep. During inference, only selected input-dependent connections are computed densely, skipping unselected ones efficiently. Experiments on Wan 2.1 14B, Mochi 1, and few-step distilled models demonstrate up to 1.58× end-to-end speedup, surpassing other training-free methods while preserving video generation quality and text-video alignment.

Key takeaway

For Machine Learning Engineers optimizing text-to-video diffusion models, you should consider implementing calibrated sparse attention techniques like CalibAtt. This training-free approach offers up to a 1.58× end-to-end speedup by intelligently skipping redundant attention computations, without compromising video quality or text-video alignment. Integrating an offline calibration step into your workflow can significantly reduce inference costs and latency for high-quality video generation.

Key insights

Text-to-video generation can be accelerated by skipping consistently negligible and repetitive spatiotemporal attention computations.

Principles

• Spatiotemporal attention includes consistently negligible token connections.
• Negligible attention patterns often repeat across queries and blocks.
• Offline calibration identifies stable sparsity for optimized inference.

Method

CalibAtt uses an offline calibration pass to identify stable block-level sparsity and repetition patterns across inputs, compiling these into optimized attention operations for inference, skipping unselected connections.

In practice

• Perform offline calibration to map stable attention sparsity.
• Compile patterns into optimized attention operations per layer.
• Skip unselected attention connections during inference.

Topics

• Text-to-Video Generation
• Diffusion Models
• Sparse Attention
• Inference Acceleration
• CalibAtt
• Spatiotemporal Attention

Best for: AI Engineer, Computer Vision Engineer, Research Scientist, AI Scientist, Machine Learning Engineer

Related on AIssential

• Accelerating Text-to-Video Generation with Calibrated Sparse Attention — CalibAtt accelerates text-to-video generation by skipping consistently negligible attention computations via calibrated sparse attention.
• Ms. Forcing: Efficient Streaming Video Generation with Multi-Scale Patchification and Attention — Efficient streaming video generation is achieved by dynamically adjusting spatial granularity based on noise levels.
• Sparse Attention for Dense Open-Vocabulary Prediction in CLIP — Sparsifying CLIP's dense attention with α-entmax at inference improves fine-grained open-vocabulary prediction by denoising irrelevant token dependencies.
• HeadCast: Casting Attention Heads for Efficient Autoregressive Video Generation — Attention heads in AR video diffusion models exhibit stable, heterogeneous behaviors that can be exploited for efficient inference.
• Sparse Forcing: Native Trainable Sparse Attention for Real-time Autoregressive Diffusion Video Generation — Sparse Forcing improves long-horizon video generation quality and efficiency by leveraging trainable sparse attention and persistent memory.
• NaviCache: Test-Time Self-Calibration Caching for Video Generation — NaviCache uses an INS-inspired dual-state estimation for error-bounded computation skipping in Video Diffusion Models.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Apple Machine Learning Research.