Qwen Team Releases FlashQLA: a High-Performance Linear Attention Kernel Library That Achieves Up to 3× Speedup

2026-04-29 · Source: Machine Learning ML & Generative AI News · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Expert, quick

Summary

The Qwen Team has released FlashQLA, a new high-performance linear attention kernel library designed to accelerate GDN (Gated Delta Network) Chunked Prefill, the linear attention mechanism used in Qwen3.5 and Qwen3.6 model families. Benchmarked against FLA 0.5.0, Triton 3.5.1, and FlashInfer 0.6.9 on NVIDIA Hopper (H200) GPUs, FlashQLA achieves a 2-3x speedup for forward passes and a 2x speedup for backward passes over the FLA Triton kernel. Its performance gains stem from three key optimizations: gate-driven automatic intra-card context parallelism, hardware-friendly algebraic reformulation to reduce overhead, and TileLang fused warp-specialized kernels that overlap data movement and computation.

Key takeaway

For AI Engineers deploying Qwen3.5 or Qwen3.6 models on NVIDIA Hopper GPUs, integrating FlashQLA can significantly reduce inference and training times. You should consider adopting this library to achieve up to 3x faster forward passes and 2x faster backward passes, directly improving model throughput and efficiency without complex manual configuration.

Key insights

FlashQLA significantly accelerates linear attention for Qwen models via specialized kernel optimizations.

Principles

• Exploit gate properties for parallelism
• Reformulate algebra for hardware efficiency
• Specialize kernels for computation overlap

Method

FlashQLA uses TileLang to implement warp-specialized kernels, enabling automatic intra-card context parallelism and algebraic reformulation to optimize GDN Chunked Prefill.

In practice

• Integrate FlashQLA for Qwen3.5/3.6 inference
• Utilize on NVIDIA Hopper GPUs for speedup

Topics

• FlashQLA
• Linear Attention
• Kernel Library
• NVIDIA Hopper
• GDN Chunked Prefill

Code references

• QwenLM/FlashQLA

Best for: AI Engineer, NLP Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, AI Hardware Engineer

Related on AIssential

• Moonshot AI Open-Sources FlashKDA: CUTLASS Kernels for Kimi Delta Attention with Variable-Length Batching and H20 Benchmarks — FlashKDA accelerates Kimi Delta Attention prefill, enhancing Moonshot AI's Kimi Linear model performance.
• Engineering Qwen-VL for Production: Vision Module Architecture and Optimization Practices — Optimizing Qwen-VL's vision module on AMD MI308X GPUs significantly boosts multimodal inference performance through kernel-level enhancements.
• FlashAttention-4: Supercharging Transformer Attention on NVIDIA Blackwell GPUs — FlashAttention-4 optimizes Transformer attention for NVIDIA Blackwell GPUs by co-designing algorithms and kernels to overcome asymmetric hardware scaling.
• 13+ Attention Mechanisms You Should Know — Attention mechanisms enable AI models to dynamically focus on relevant tokens, crucial for understanding sequence context and meaning.
• Vortex: Efficient and Programmable Sparse Attention Serving for AI Agents — Vortex simplifies sparse attention algorithm development and deployment by abstracting complex paged memory layouts.
• Qwen3-Coder-Next: How to Run Qwen’s 80B Sparse MoE Coder — Qwen3-Coder-Next uses sparse MoE and hybrid attention to enable efficient, long-context coding agent capabilities.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning ML & Generative AI News.