A JoLT for the KV Cache: Near-Lossless KV Cache Compression via Joint Tucker and JL-Residual Allocation for LLMs

2026-07-15 · Source: cs.CL updates on arXiv.org · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Expert, extended

Summary

JoLT and FlashJoLT introduce a novel KV-cache compression method for large language models, addressing the memory bottleneck in transformer inference. JoLT treats the cache as a third-order tensor, applying a partial Tucker decomposition to compress only the token and feature axes, which carry most redundancy. It then restores discarded energy using a Johnson-Lindenstrauss (JL) rotated low-bit residual. A single Lagrangian dual jointly allocates Tucker ranks and residual bit-widths per layer group and separately for keys and values, under a byte budget. This achieves near-lossless 2–3× compression, maintaining perplexity, GSM8K accuracy, and RULER retrieval within statistical noise on Mistral-7B-v0.3 (GQA) and LLaMA-2-13B (MHA). At 2×, JoLT reconstructs the cache with relative Frobenius errors of 0.009 (K) and 0.006 (V). FlashJoLT, a randomized-SVD variant, further accelerates compression time by 5–13× at matched quality.

Key takeaway

For MLOps Engineers optimizing LLM inference costs, you should implement JoLT or FlashJoLT to achieve 2–3× KV cache compression. This significantly reduces memory footprint for long contexts without impacting model quality on perplexity or downstream tasks. Be aware that Multi-Head Attention models like LLaMA-2-13B degrade sharply beyond 4× compression, so stick to the 2–3× "free zone" for robust performance. Consider fused kernels for decode-time efficiency.

Key insights

KV cache compression is near-lossless at 2–3× by treating it as a tensor and jointly allocating ranks and residual bits.

Principles

• KV cache redundancy is primarily in token and feature axes.
• Values are 2–3× harder to compress than keys.
• Joint allocation of ranks and residual bits is crucial.

Method

JoLT uses partial Tucker decomposition on token/feature axes, then a JL-rotated low-bit residual, with a Lagrangian dual for joint rank/bit allocation.

In practice

• Target 2–3× KV cache compression for LLMs.
• Use randomized SVD for faster compression.
• Prioritize residual bits for value compression.

Topics

• KV Cache Compression
• Tensor Decomposition
• LLM Inference
• FlashJoLT
• Mistral-7B
• LLaMA-2-13B

Best for: AI Engineer, NLP Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, MLOps Engineer

Related on AIssential

• A JoLT for the KV Cache: Near-Lossless KV Cache Compression via Joint Tucker and JL-Residual Allocation for LLMs — JoLT compresses LLM KV caches by treating them as third-order tensors, combining Tucker decomposition with JL-residual allocation for near-lossless results.
• Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving — KV-cache optimization effectiveness is workload-dependent, with stable quantization balancing quality and performance.
• Fractal KV-Cache Archives: Lossless Symbolic Storage with In-Place Retrieval for Long-Context LLM Inference — Fractal KV-Cache Archives offer lossless, O(1) access storage for quantized LLM KV caches, doubling as a search index.
• Fast KVzip: Much Faster KV Cache Eviction for Cheaper Inference — Causal attention in decoder-only LLMs limits context utilization when options precede context.
• Rotate, Then Round: The Geometry of KV-Cache Compression — KV-cache compression evolved from simple rounding to geometric transformations, but production revealed that unbiasedness isn't enough when softmax amplifies variance.
• VarRate: Training-Free Variable-Rate KV Cache Compression for Long-Context LLMs — VarRate compresses KV cache by assigning variable low-rank budgets based on query salience, retaining all tokens.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by cs.CL updates on arXiv.org.