Moonshot AI Open-Sources FlashKDA: CUTLASS Kernels for Kimi Delta Attention with Variable-Length Batching and H20 Benchmarks

2026-05-01 · Source: Machine Learning ML & Generative AI News · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, quick

Summary

Moonshot AI has open-sourced FlashKDA, a set of CUTLASS kernels designed to accelerate Kimi Delta Attention (KDA), the core mechanism in their Kimi Linear hybrid model. FlashKDA significantly improves prefill speed, achieving 1.72x to 2.22x faster performance than the flash-linear-attention baseline on NVIDIA H20 GPUs. KDA itself reduces KV cache usage by up to 75% and boosts decoding throughput by up to 6x at 1M context length by replacing traditional attention's expanding KV cache with a fixed-size recurrent state. FlashKDA supports variable-length batching via `cu_seqlens` and is auto-dispatched from `flash-linear-attention's chunk_kda`, requiring zero code changes. It is released under an MIT license, compatible with SM90+, CUDA 12.9+, and PyTorch 2.4+.

Key takeaway

For AI engineers optimizing large language model inference, FlashKDA offers a substantial performance boost for Kimi Delta Attention. If your projects utilize Kimi Linear or similar attention mechanisms, integrating FlashKDA can significantly reduce prefill times and improve overall throughput, especially on NVIDIA H20 GPUs. Consider adopting this MIT-licensed solution to enhance your model's efficiency without extensive code modifications.

Key insights

FlashKDA accelerates Kimi Delta Attention prefill, enhancing Moonshot AI's Kimi Linear model performance.

Principles

• Fixed-size recurrent states reduce KV cache usage.
• Optimized kernels improve prefill and decoding throughput.

Method

FlashKDA utilizes CUTLASS kernels to accelerate Kimi Delta Attention, supporting variable-length batching and integrating seamlessly with existing `flash-linear-attention` implementations.

In practice

• Integrate FlashKDA for faster KDA prefill.
• Leverage variable-length batching for efficiency.

Topics

• FlashKDA
• Kimi Delta Attention
• CUTLASS Kernels
• Variable-Length Batching
• NVIDIA H20

Code references

• MoonshotAI/FlashKDA

Best for: AI Engineer, NLP Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, AI Hardware Engineer

Related on AIssential

• Qwen Team Releases FlashQLA: a High-Performance Linear Attention Kernel Library That Achieves Up to 3× Speedup — FlashQLA significantly accelerates linear attention for Qwen models via specialized kernel optimizations.
• Kimi K2.7-Code cuts thinking tokens 30% — but practitioners say the benchmarks don't check out — Kimi K2.7-Code offers claimed token efficiency and proprietary benchmark gains, but independent tests reveal mixed performance and raise benchmark transparency concerns.
• Build with Kimi K2.5 Multimodal VLM Using NVIDIA GPU-Accelerated Endpoints — Kimi K2.5 is a multimodal VLM optimized for agentic AI and diverse tasks, built on Megatron-LM with a specialized vision tower.
• KV Cache Demystified: Speeding Up Large Language Models — KV cache optimizes LLM inference by storing and reusing attention keys and values, preventing redundant computations.
• Building and Training a Kimi-K2 Model Using DeepSeek-V3 Components — Kimi-K2 optimizes agentic LLM performance through architectural sparsity, a novel optimizer, and enhanced training data utility.
• Sakana AI and NVIDIA Introduce TwELL with CUDA Kernels for 20.5% Inference and 21.9% Training Speedup in LLMs — TwELL and new CUDA kernels exploit LLM sparsity for significant speedups and energy savings on NVIDIA GPUs.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Machine Learning ML & Generative AI News.