The Architecture of Kimi Delta Attention

2026-07-26 · Source: LLM on Medium · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Emerging Technologies & Innovation · Depth: Expert, short

Summary

Kimi Delta Attention (KDA), introduced in Moonshot AI's Kimi K3 model architecture, addresses the O(N) memory bottleneck of standard Transformer KV caches when processing massive contexts exceeding 100,000 tokens, up to 1,000,000 tokens. KDA replaces the unbounded KV cache with a fixed-size 128x128 state matrix per attention head, maintaining static memory utilization regardless of context length. It achieves this by combining per-channel diagonal decay gates, which allow different features to fade at varying speeds, and the Delta Rule overwrite mechanism, which explicitly subtracts stale information before adding new values to prevent "memory tie" bugs. The Kimi K3 architecture further enhances this by interleaving three KDA layers with one full Multi-Head Latent Attention (MLA) layer. This hybrid design results in up to a 75% reduction in total KV cache memory footprint compared to all-attention architectures, while preserving near-identical performance on complex benchmarks.

Key takeaway

For AI Architects designing Large Language Models for extreme long-context applications, Kimi Delta Attention offers a viable path to overcome VRAM bottlenecks. You should consider integrating fixed-memory linear attention mechanisms with explicit overwrite rules and per-channel decay to manage context windows up to 1,000,000 tokens. This approach can significantly reduce your KV cache memory footprint by up to 75% while maintaining performance, enabling more efficient inference at scale.

Key insights

Kimi Delta Attention uses fixed-size memory and explicit overwriting to scale LLMs to 1M+ token contexts efficiently.

Principles

• Fixed-size memory prevents O(N) scaling.
• Explicit overwriting resolves memory conflicts.
• Per-channel decay enables selective forgetting.

Method

KDA updates a fixed-size state matrix by reading existing data, subtracting stale values via the Delta Rule, adding new values, and applying per-channel diagonal decay for selective forgetting.

In practice

• Implement fixed-size state matrices for long contexts.
• Use Delta Rule for explicit memory updates.
• Apply per-channel decay for nuanced information retention.

Topics

• Kimi Delta Attention
• Large Language Models
• Long Context Windows
• KV Cache Optimization
• Linear Attention
• Memory Management

Best for: Research Scientist, AI Engineer, NLP Engineer, AI Scientist, Machine Learning Engineer, AI Architect

Related on AIssential

• THE ARCHITECTURE OF PERMANENCE: DECONSTRUCTING KIMI-K3 — Kimi K3's "architecture of permanence" uses Attention Residuals and TOPO-2026 to mathematically guarantee against catastrophic forgetting.
• KV Cache Demystified: Speeding Up Large Language Models — KV cache optimizes LLM inference by storing and reusing attention keys and values, preventing redundant computations.
• Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing — Comparative study reveals Kimi Delta Attention with Muon and Cross-Layer Value Routing (CLVR) improve linear attention performance.
• Moonshot AI Open-Sources FlashKDA: CUTLASS Kernels for Kimi Delta Attention with Variable-Length Batching and H20 Benchmarks — FlashKDA accelerates Kimi Delta Attention prefill, enhancing Moonshot AI's Kimi Linear model performance.
• A Hippocampus for Linear Attention: An Exact Memory for What the Recurrent State Forgets — HOLA enhances linear attention models with an exact KV cache, preventing memory loss and improving long-context recall.
• Building and Training a Kimi-K2 Model Using DeepSeek-V3 Components — Kimi-K2 optimizes agentic LLM performance through architectural sparsity, a novel optimizer, and enhanced training data utility.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by LLM on Medium.