TurboQuant end to end

Putting the pieces together on many key vectors x from a KV cache, with b bits per coordinate. TurboQuantmse rotates, quantizes every coordinate with the b-bit Lloyd–Max codebook, and rotates back. It minimizes ‖x − x̃‖2, but its inner products ⟨y, x̃⟩ are biased toward zero. TurboQuantprod spends b − 1 bits on TurboQuantmse and the last bit on QJL applied to the residual r = x − x̃mse, which makes ⟨y, x̃⟩ an unbiased estimate of ⟨y, x⟩ for every query y.

Over many keys and queries

Random key/query pairs (keys with outlier channels, normalized to unit length; queries at random angles to them). Each dot is one pair, quantized with its own random Π and S: TurboQuant's guarantees are expectations over that randomness. (In a real cache one Π and S are shared, and across many different keys the errors average out the same way; our synthetic keys are all dominated by the same outlier channels, so with one shared draw they would mostly see the same luck, good or bad.)

TurboQuant mse

TurboQuant prod

TurboQuant mse errors

TurboQuant prod errors

All bit widths

Slope: least-squares fit of the estimate against the true ⟨y, x⟩ (1 = unbiased). Paper's upper bounds on E‖x − x̃‖² for TurboQuantmse: √3π/2 · 4−b ≈ 0.68, 0.17, 0.043, 0.011, with actual values ≈ 0.36, 0.117, 0.03, 0.009. For TurboQuantprod at b = 1 the whole budget goes to QJL.

Inner-product residuals ⟨y, x̃⟩ − ⟨y, x⟩

The true inner products are spread symmetrically around 0, so shrinkage toward zero cancels in the plain mean; the "toward 0" column multiplies each residual by sign⟨y, x⟩, so negative values mean the estimates are pulled toward zero (bias). TurboQuantmse is biased but has lower variance; TurboQuantprod is unbiased but has higher variance.

Which variant is used in practice?

Mostly TurboQuantmse. The paper's downstream experiments (needle-in-a-haystack, LongBench, near-neighbor search) use TurboQuantprod but never compare it with TurboQuantmse at the same bit budget, and the paper's own Fig. 2a shows prod winning on inner-product error only at low bit widths. Practitioners have generally dropped the QJL stage: vLLM's TurboQuant backend omits it because it was found to "hurt attention quality by amplifying variance through softmax", and llama.cpp implementers report the same. A perplexity comparison on Qwen2.5-7B reproduced in HyperQuant finds the MSE variant better at 2, 3 and 4 bits. Unbiasedness removes the shrinkage of attention scores, but the extra variance it costs (the wider histogram above) matters more.