Looped Latent Attention: Cross-Loop KV Compression for Looped Transformers

2026-07-16 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Natural Language Processing · Depth: Expert, quick

Summary

Looped Latent Attention (LLA) is a novel post-training cache codec designed for Looped Transformers, which typically reuse a block to reduce parameters but still store a separate K/V cache for each recurrence step. LLA addresses this by exploiting the low-rank trajectory of K/V vectors across loops, storing compact K and V latents and reconstructing loop-specific vectors only when needed. The system, initialized via SVD and refined with KL and attention-output distillation, offers a per-head codec for recurrence compression and an LLA-2D variant for extreme compression by folding heads. LLA outperforms alternative cache reduction methods like head-axis MLA and KV quantization at equivalent cache budgets. It achieves near-lossless 32x compression on Huginn-3.5B and increases Ouro-1.4B batch capacity from 32 to 768 sequences (21.3x compression) at 4k context on an H200. Furthermore, on-policy refinement improves MATH-500 scores from 0.43 to 0.66.

Key takeaway

For Machine Learning Engineers optimizing inference for recurrent Transformer models, Looped Latent Attention (LLA) presents a critical method to drastically reduce K/V cache memory. You should consider implementing LLA to achieve up to 32x cache compression, significantly increasing batch capacity—e.g., from 32 to 768 sequences for Ouro-1.4B at 4k context on an H200. This directly improves throughput and can enhance performance on long context tasks like math rollouts by reducing errors and boosting accuracy.

Key insights

Looped Latent Attention compresses K/V caches in Looped Transformers by exploiting low-rank recurrence, significantly boosting inference efficiency.

Principles

• K/V vectors in looped Transformers exhibit low-rank trajectories across recurrence steps.
• Recurrent K/V caches are low-rank but not safely collapsible to a single state.
• Distillation and SVD can effectively compress attention caches post-training.

Method

LLA uses SVD-initialized and KL/attention-output distilled codecs to store compact K/V latents, reconstructing loop-specific vectors on demand.

In practice

• Increase Ouro-1.4B batch capacity from 32 to 768 at 4k context.
• Achieve 32x K/V cache compression for models like Huginn-3.5B.
• Improve MATH-500 scores for long math rollouts via on-policy refinement.

Topics

• Looped Transformers
• K/V Cache Compression
• Latent Attention
• SVD Codec
• Inference Optimization
• Batch Capacity
• MATH-500 Benchmark

Best for: MLOps Engineer, AI Engineer, NLP Engineer, AI Scientist, Machine Learning Engineer, AI Architect

Related on AIssential

• Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attention — LLM architectures are evolving with complex, targeted tweaks to optimize long-context efficiency and reduce computational overhead.
• Looped Transformer !! (And More) — Looped Transformers enhance reasoning and efficiency by iteratively applying the same layer, scaling compute at test time.
• Moment-KV: Momentum-Based Decode-Time KV Cache Compression for Long Generation — Moment-KV improves LLM long-generation fidelity by using momentum-driven attention aggregation for decoding-time KV cache compression.
• MoA-Structured Decode Attention DNF Derivation, KV-Cache Accumulation, GQA/MQA, and OpenACC Kernel — MoA enables memory-optimal transformer attention inference through algebraic derivation and efficient kernel implementation.
• VideoMLA: Low-Rank Latent KV Cache for Minute-Scale Autoregressive Video Diffusion — VideoMLA significantly reduces KV cache memory in video diffusion by using low-rank latent attention, improving long-horizon performance and throughput.
• REAL: REtrieval-reAsoning and Logic-constructed Attention Behaviors for Long-Context KV Cache Compression — LLM KV cache compression can be robustly optimized by distinguishing and allocating resources based on "surface memorization" and "logic construction" attention behaviors.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.