The KV cache is a major memory bottleneck for long-context LLM inference. Traditional vector quantization methods like Product Quantization require expensive dataset-specific codebook training that can take hundreds of seconds. TurboQuant, from Google Research (ICLR 2026), is data-oblivious: no training, no calibration, works instantly.
The key insight: applying a random rotation to input vectors induces a concentrated Beta distribution on each coordinate in high dimensions, making coordinates nearly i.i.d. This lets you solve a simple 1D scalar quantization problem per coordinate instead of a complex joint optimization. Codebooks are precomputed once per bit-width and reused at inference time.
For attention mechanisms, MSE-optimal quantization introduces bias in inner product estimation. TurboQuant fixes this with a two-stage approach: quantize at b-1 bits for MSE, then apply a 1-bit Quantized Johnson-Lindenstrauss transform to the residual. The result is a provably unbiased inner product estimator at b total bits.
Numbers: 6x KV cache compression with 100% retrieval accuracy on Needle-In-A-Haystack up to 104k tokens. 4-bit TurboQuant delivers up to 8x speedup over 32-bit unquantized keys on H100. Indexing time for 1536-dimensional vectors: 0.0013s vs 239s for Product Quantization.