A JoLT for the KV Cache: Near-Lossless KV Cache Compression via Joint Tucker and JL-Residual Allocation for LLMs

2026-07-14 · Source: Computation and Language · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Mathematics & Computational Sciences · Depth: Expert, quick

Summary

JoLT introduces a novel method for near-lossless key-value (KV) cache compression, addressing the dominant memory cost in Transformer inference. This technique views the cache at a layer as a third-order tensor, exploiting varying redundancy across its heads, tokens, and features axes. JoLT applies a partial Tucker decomposition to compress the token and feature axes, then restores discarded energy using a Johnson-Lindenstrauss (JL) rotated low-bit residual. A single Lagrangian dual jointly allocates Tucker ranks and residual bit-widths per layer group. This achieves 2-3x compression, maintaining perplexity, GSM8K accuracy, and RULER retrieval within statistical noise on Mistral-7B-v0.3 and LLaMA-2-13B. At 2x compression, it yields relative Frobenius errors of 0.009 (K) and 0.006 (V). FlashJoLT offers a 5-13x compression-time speedup.

Key takeaway

For Machine Learning Engineers optimizing large language model inference, JoLT offers a compelling solution to reduce KV cache memory footprint. You can achieve a near-lossless 2-3x compression, maintaining model performance on benchmarks like GSM8K and RULER, as demonstrated on Mistral-7B-v0.3 and LLaMA-2-13B. Consider integrating JoLT or its faster variant, FlashJoLT, to significantly improve throughput and reduce GPU memory requirements for long-context LLM deployments.

Key insights

JoLT compresses LLM KV caches by treating them as third-order tensors, combining Tucker decomposition with JL-residual allocation for near-lossless results.

Principles

• KV cache is the dominant memory cost in Transformer inference.
• Cache at a layer is a third-order tensor with varying redundancy.
• Joint allocation of compression methods optimizes byte budget.

Method

JoLT applies partial Tucker decomposition on token and feature axes, then restores energy with a Johnson-Lindenstrauss (JL) rotated low-bit residual, allocated via a Lagrangian dual per layer group.

In practice

• Achieve 2-3x KV cache compression with minimal performance impact.
• Use FlashJoLT for 5-13x faster compression at similar quality.

Topics

• KV Cache Compression
• Transformer Inference
• Tucker Decomposition
• Johnson-Lindenstrauss
• LLM Memory Optimization
• Mistral-7B
• LLaMA-2

Best for: AI Engineer, Research Scientist, AI Scientist, Machine Learning Engineer, NLP Engineer

Related on AIssential

• A JoLT for the KV Cache: Near-Lossless KV Cache Compression via Joint Tucker and JL-Residual Allocation for LLMs — KV cache compression is near-lossless at 2–3× by treating it as a tensor and jointly allocating ranks and residual bits.
• Rotate, Then Round: The Geometry of KV-Cache Compression — KV-cache compression evolved from simple rounding to geometric transformations, but production revealed that unbiasedness isn't enough when softmax amplifies variance.
• KVpop -- Key-Value Cache Compression with Predictive Online Pruning — KVpop uses future-attention supervision and delayed stateful scoring to predict token utility for efficient, fixed-budget KV cache compression.
• PolyKV: Heterogeneous Retention and Allocation for KV Cache Compression — PolyKV optimizes KV cache compression by applying heterogeneous, layer-wise policies and non-uniform budget allocation, significantly improving long-context LLM performance.
• Fast KVzip: Much Faster KV Cache Eviction for Cheaper Inference — Causal attention in decoder-only LLMs limits context utilization when options precede context.
• KV Cache Demystified: Speeding Up Large Language Models — KV cache optimizes LLM inference by storing and reusing attention keys and values, preventing redundant computations.

See Counsel's argued verdicts on the open AI decisions leaders are weighing →

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by Computation and Language.