TurboQuant: ~3-bit KV Cache with Near 0 Accuracy Loss?

2026-04-13 · Source: The Kaitchup – AI on a Budget · Field: Technology & Digital — Artificial Intelligence & Machine Learning, Software Development & Engineering · Depth: Advanced, short

Summary

TurboQuant is a KV-cache quantization method designed to reduce memory consumption and accelerate decoding in large language model (LLM) inference, particularly for long sequences. The technique, initially proposed in a one-year-old paper and popularized by Google, applies a fixed orthogonal rotation to KV vectors to make them more amenable to low-bit quantization, then stores them in a compact format. This approach aims to transform difficult-to-quantize KV vectors into a statistically regular form, allowing a fixed scalar codebook to perform near-optimally. The article evaluates TurboQuant's accuracy using two implementations, vLLM and llama.cpp, focusing on whether it degrades the quality of generated sequences rather than speed or memory benchmarks. Experiments with vLLM using a 2-bit key and 4-bit value configuration showed no significant degradation on HumanEval and GPQA Diamond benchmarks, despite vLLM's unstable support for the feature.

Key takeaway

For AI Engineers optimizing LLM inference for long contexts, TurboQuant offers a promising approach to reduce KV cache memory footprint and potentially increase throughput. Your teams should consider experimenting with TurboQuant's `tq3` (2-bit key, 4-bit value) configuration in vLLM, noting that while accuracy appears preserved, current vLLM integration may be unstable. Prioritize testing on challenging, long-context tasks like GPQA Diamond to assess real-world impact on reasoning traces.

Key insights

TurboQuant uses orthogonal rotation and low-bit quantization to compress KV caches, improving LLM inference efficiency without significant accuracy loss.

Principles

• KV cache size limits LLM inference.
• Orthogonal rotations simplify vector quantization.
• Fixed scalar codebooks can be effective post-rotation.

Method

TurboQuant applies a fixed random orthogonal rotation to KV vectors, quantizes each rotated coordinate to a precomputed codebook, and reconstructs via inverse rotation and codebook lookup.

In practice

• Configure vLLM with `--kv-cache-dtype tq3` for 2-bit MSE.
• Set `TQ_VALUE_BITS=4` for 4-bit value precision.
• Expect ~4x KV cache size reduction with tq3.

Topics

• KV Cache Quantization
• TurboQuant Algorithm
• LLM Inference Optimization
• Orthogonal Transforms
• vLLM Integration

Code references

• vllm-project/vllm
• vllm-project/vllm

Best for: Machine Learning Engineer, AI Engineer, AI Scientist

Related on AIssential

• TurboQuant: Finally, Fast and Widely Available Low-Bit KV Cache Quantization? — TurboQuant uses random orthogonal rotation and scalar codebooks for efficient, training-free KV cache quantization.
• Google’s TurboQuant Compression May Support Faster Inference, Same Accuracy on Less Capable Hardware — TurboQuant compresses LLM KV caches up to 6x with minimal accuracy loss, enabling long contexts on less capable hardware.
• Productionizing TurboQuant on AMD GPUs for KV-Cache-Bound LLM Inference — Optimized TurboQuant on AMD GPUs significantly boosts LLM inference throughput and reduces latency for KV-cache-bound agentic workloads.
• TurboQuant: Redefining AI efficiency with extreme compression — TurboQuant offers extreme compression for LLMs and vector search with zero accuracy loss and significant speedups.
• KV Cache Is Eating Your VRAM. Here’s How Google Fixed It With TurboQuant. — TurboQuant offers near-optimal KV cache compression for LLMs, preserving accuracy by focusing on attention mechanism needs.
• PolarQuant: Leveraging Polar Transformation for Efficient Key Cache Quantization and Decoding Acceleration — PolarQuant leverages polar coordinate transformation to efficiently quantize LLM key caches, mitigating outliers and accelerating decoding via table lookups.

Open in AIssential →

Editorial summary, takeaway, and curation by AIssential. Original article published by The Kaitchup – AI on a Budget.