By Amir Zandieh, Majid Daliri, Majid Hadian, Vahab MirrokniSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2504-19874.git
Cite
Version link
https://openversions.com/index/arxiv-2504-19874/commit/cdda9964231f7b5ffe80c02a00ea94b344a7ff30
Version identifier (commit hash)
cdda9964231f7b5ffe80c02a00ea94b344a7ff30

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Latest version

Checks complete

Archive from arXiv Source: https://arxiv.org/abs/2504.19874v1 OpenVersions: 93533b48-febd-4ad3-ae13-bce8ed4bac7b

Posted

Claims

Findings

Headline finding

TurboQuant is a data-oblivious, online vector quantization framework that achieves near-optimal distortion rates, within a small constant factor (approximately 2.7) of the information-theoretic lower bound, across all bit-widths and dimensions for both mean-squared error (MSE) and inner product distortion.

For any bit-width b≥1b \ge 1 and any worst-case unit vector x∈Sd−1\boldsymbol{x} \in \mathbb{S}^{d-1}, the MSE-optimized TurboQuant (QmseQ_{\text{mse}}) achieves an expected MSE distortion bounded by Dmse≤3π2⋅14bD_{\text{mse}} \le \frac{\sqrt{3\pi}}{2} \cdot \frac{1}{4^b}, with finer-grained values Dmse≈0.36,0.117,0.03,0.009D_{\text{mse}} \approx 0.36, 0.117, 0.03, 0.009 for b=1,2,3,4b = 1, 2, 3, 4, respectively.

For any bit-width b≥1b \ge 1, any x∈Sd−1\boldsymbol{x} \in \mathbb{S}^{d-1}, and any y∈Rd\boldsymbol{y} \in \mathbb{R}^d, the inner-product optimized TurboQuant (QprodQ_{\text{prod}}) provides an unbiased inner product estimate Ex~[⟨y,x~⟩]=⟨y,x⟩\mathbb{E}_{\tilde{\boldsymbol{x}}}[\langle \boldsymbol{y}, \tilde{\boldsymbol{x}} \rangle] = \langle \boldsymbol{y}, \boldsymbol{x} \rangle and achieves inner-product distortion Dprod≤3π2⋅∥y∥22d⋅14bD_{\text{prod}} \le \frac{\sqrt{3}\pi^2 \cdot \|\boldsymbol{y}\|_2^2}{d} \cdot \frac{1}{4^b}, with finer-grained values Dprod≈1.57d,0.56d,0.18d,0.047dD_{\text{prod}} \approx \frac{1.57}{d}, \frac{0.56}{d}, \frac{0.18}{d}, \frac{0.047}{d} for b=1,2,3,4b = 1, 2, 3, 4, respectively.

For any randomized quantization algorithm Q:Sd−1→{0,1}b⋅dQ: \mathbb{S}^{d-1} \to \{0,1\}^{b \cdot d} with bit-width bb and reconstruction map Q−1Q^{-1}, there exist hard input instances x,y∈Sd−1\boldsymbol{x}, \boldsymbol{y} \in \mathbb{S}^{d-1} such that Dmse(Q)≥14bD_{\text{mse}}(Q) \ge \frac{1}{4^b} and Dprod(Q)≥1d⋅14bD_{\text{prod}}(Q) \ge \frac{1}{d} \cdot \frac{1}{4^b}.

For KV cache quantization, TurboQuant achieves absolute quality neutrality with 3.5 bits per channel and marginal quality degradation with 2.5 bits per channel, achieving identical performance to the full-precision baseline on Needle-in-a-Haystack at 4x compression and outperforming competing methods on LongBench-E.

In high-dimensional nearest neighbor search tasks, TurboQuant consistently outperforms Product Quantization (PQ) and RabitQ in recall@1@k while reducing quantization/indexing time to virtually zero.

Methods

TurboQuant_mse: Quantizes a vector by multiplying it with a random rotation matrix (generated via QR decomposition of a Gaussian matrix) and applying an optimal 1D Lloyd-Max scalar quantizer (precomputed for the induced Beta distribution) independently to each coordinate.

Source excerpts

“Therefore the quantizer Qmse : R^d → {0, 1}^{b·d} first computes Π · x and then computes and stores the indices of the nearest centroids to each coordinate of this vector. The dequantization map Q−1Q^{−1}_mse : {0, 1}^{b·d} → R^d reconstructs the vector by retrieving the centroids corresponding to the stored indices and then rotating the result back to the original basis through multiplication with Π^⊤.” · Show in paper

TurboQuant_prod: A two-stage inner-product quantizer that applies TurboQuant_mse at bit-width b-1, computes the residual vector, and applies a 1-bit Quantized Johnson-Lindenstrauss (QJL) transform to the residual while storing the residual's L2 norm.

Source excerpts

“Specifically, let Qmse be the quantization map corresponding to TurboQuantmse with a bit-width of b − 1. For any x ∈ Sd−1S^{d−1} the residual vector, defined as r := x − Q−1Q^{−1}_mse (Qmse(x)), has a small L2 norm... We can then apply the QJL quantization map Qqjl on this residual vector, resulting in an overall bit-width of b” · Show in paper

Outlier-aware channel splitting for KV cache quantization: Splits channels into outlier and non-outlier sets and applies two independent instances of TurboQuant with higher bit precision allocated to outlier channels to achieve fractional effective bit-widths such as 2.5 and 3.5 bits.

Source excerpts

“These non-integer bit precisions result from our strategy of splitting channels into outlier and non-outlier sets, and applying two independent instances of TurboQuant to each, allocating higher bit precision to outliers.” · Show in paper