HeadCast: Casting Attention Heads for Efficient Autoregressive Video Generation

2026-07-23 · Source: cs.CV updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Computer Vision, Software Development & Engineering · Depth: Expert, extended

Summary

HeadCast is a training-free, plug-and-play acceleration framework for autoregressive (AR) video diffusion models, addressing the dominant inference cost of attention due to growing Key-Value (KV) caches. It operates by observing that pre-trained AR models' attention heads exhibit stable, heterogeneous behaviors. After a brief warm-up, HeadCast performs a one-time classification at the maximum-noise step ($t=1000$), sorting each head into one of four archetypes: Sink, Dummy, Spatial, or Global. This allows restructuring the monolithic KV cache into head-specific pathways, crucially retaining Global heads for long-range temporal consistency. The Spatial pathway's fixed-size grid ensures savings scale with resolution, accelerating inference by up to 1.62$\times$ at 720P and 1.95$\times$ at 1080P across state-of-the-art AR models like Self-Forcing and LongLive, while maintaining VBench quality and largely flicker-free output. Over 90% of head assignments remain stable across autoregressive steps.

Key takeaway

For Machine Learning Engineers optimizing autoregressive video generation, HeadCast offers a training-free solution to significantly reduce inference costs without sacrificing visual fidelity. You should consider integrating this plug-and-play framework to achieve up to 1.95$\times$ speedup at high resolutions. This approach preserves temporal consistency, avoiding the flickering issues common with aggressive cache eviction methods.

Key insights

Attention heads in AR video diffusion models exhibit stable, heterogeneous behaviors that can be exploited for efficient inference.

Principles

• Attention head archetypes (Sink, Dummy, Spatial, Global) are stable.
• Classify heads once at maximum noise for optimal fidelity.
• Retaining Global heads prevents temporal flickering.

Method

HeadCast performs a warm-up, then a one-time classification at $t=1000$ using cosine similarity and a spatial score to assign heads to Sink, Dummy, Spatial, or Global archetypes, then routes them to tailored KV cache pathways.

In practice

• Implement head-specific KV cache management for AR video models.
• Classify attention heads early in the denoising process.
• Prioritize retaining global context for temporal coherence.

Topics

• Autoregressive Video Generation
• Diffusion Models
• Attention Mechanisms
• KV Cache Optimization
• Inference Acceleration
• Temporal Consistency
• HeadCast

Code references

• sjlgaga/HeadCast
• sjlgaga/HeadCast2

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, Computer Vision Engineer

Related on AIssential

• HeadCast: Casting Attention Heads for Efficient Autoregressive Video Generation — HeadCast accelerates AR video generation by classifying attention heads and optimizing KV cache access without retraining.
• Closing the Loop: Training-Free Revisit Consistency for Autoregressive Generative Rendering — Training-free geometric and temporal correspondence improves autoregressive generative rendering consistency during camera revisits.
• Ms. Forcing: Efficient Streaming Video Generation with Multi-Scale Patchification and Attention — Efficient streaming video generation is achieved by dynamically adjusting spatial granularity based on noise levels.
• TetherCache: Stabilizing Autoregressive Long-Form Video Generation with Gated Recall and Trusted Alignment — TetherCache stabilizes long-form video generation by intelligently managing cache and aligning historical context.
• StreamHOI: Interaction-aware Temporal Memory Adaptation for Streaming HOI Video Generation — StreamHOI adapts Transformer block memory based on spatial attention bias for consistent, low-latency, long-duration HOI video generation.
• Accelerating Text-to-Video Generation with Calibrated Sparse Attention — Text-to-video generation can be accelerated by skipping consistently negligible and repetitive spatiotemporal attention computations.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CV updates on arXiv.org.