FVAttn: Adaptive Sparse Attention with Runtime Load Balancing for Video Generation

2026-07-17 · Source: Computer Vision and Pattern Recognition · Field: Technology & Digital — Artificial Intelligence & Machine Learning · Depth: Expert, quick

Summary

FVAttn is a training-free sparse-attention system designed to enhance the distributed execution efficiency of adaptive sparse attention within Video Diffusion Transformers, specifically addressing the self-attention bottleneck in high-resolution video generation. It tackles the rank-level straggler problem caused by uneven per-head workloads under multi-GPU sequence parallelism. The system employs Top-$p$ routing, a Top-$k$ safety floor, and video-aware block organization as its sparse-routing frontend, then dynamically repairs the materialized mask at runtime. Key features include Runtime Load Balancing, which migrates heavy heads via P2P communication to shorten critical paths, and Slack-Aware Sparse Augmentation, which utilizes residual non-critical-rank slack for additional high-value blocks. On step-distilled Wan2.2 I2V, FVAttn reduces average load imbalance from 1.34 to 1.08, achieving a \$4.41\times$ attention speedup over FlashAttention and a \$2.02$--\$2.11\times$ DiT inference speedup with competitive video quality.

Key takeaway

For Machine Learning Engineers optimizing Video Diffusion Transformers on multi-GPU systems, FVAttn offers a critical solution to the self-attention bottleneck. You should consider integrating training-free adaptive sparse attention systems that incorporate runtime load balancing and slack augmentation. This approach can significantly reduce load imbalance from 1.34 to 1.08 and deliver over $2\times$ DiT inference speedups, directly impacting the feasibility and cost-efficiency of high-resolution video generation.

Key insights

FVAttn optimizes sparse attention in video diffusion models by dynamically balancing workloads across GPUs, significantly boosting inference speed.

Principles

• Adaptive sparse attention creates workload imbalance.
• Runtime load balancing can mitigate stragglers.
• Augmenting slack improves resource utilization.

Method

FVAttn uses Top-$p$ routing with a Top-$k$ safety floor and video-aware block organization, then repairs the mask at runtime. It applies Runtime Load Balancing and Slack-Aware Sparse Augmentation.

In practice

• Reduce attention cost in video generation.
• Improve multi-GPU sparse attention efficiency.
• Speed up DiT inference by over $2\times$.

Topics

• Video Diffusion Transformers
• Sparse Attention
• Multi-GPU Parallelism
• Load Balancing
• Video Generation
• DiT Inference

Best for: Computer Vision Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, AI Engineer

Related on AIssential

• Veda: Scalable Video Diffusion via Distilled Sparse Attention — Generation quality in sparse attention depends on mask alignment with full attention's tile-wise geometry, not just sparsity ratio.
• Real-Time AttentionBender: Granular Interactive Network Bending of Video Diffusion Transformers — Real-Time AttentionBender offers granular, interactive control over Video Diffusion Transformers, enhancing creative agency and model understanding.
• DiTango: Cost-Effective Parallel Diffusion Generation with Selective Attention State Reuse — DiTango optimizes parallel Diffusion Transformer generation by selectively reusing attention states, significantly reducing multi-node communication overhead.
• You Might Not Need 50 Diffusion Steps — Ziv Ilan, Nvidia — Optimizing diffusion models for real-time generation requires combining quantization, caching, and step distillation to reduce latency and compute.
• SparVAR: Exploring Sparsity in Visual AutoRegressive Modeling for Training-Free Acceleration — SparVAR accelerates Visual AutoRegressive models by exploiting attention sparsity for training-free, high-resolution image generation.
• MobileWan: Closing the Quality Gap for Mobile Video Diffusion — Server-scale 5B-parameter video diffusion models can run on mobile via recurrent reformulation and structured compression.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computer Vision and Pattern Recognition.