By Yu Li, Binxu Li, Tian LanSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2606-01563.git
Cite
Version link
https://openversions.com/index/arxiv-2606-01563/commit/3a482c6268f44273d13ee2418b9946cc5bf1fe10
Version identifier (commit hash)
3a482c6268f44273d13ee2418b9946cc5bf1fe10

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Latest version

Checks complete

Archive from arXiv Source: https://arxiv.org/abs/2606.01563v1 OpenVersions: cf6bd333-9b51-4c63-95fc-9ceeacc4ab8e

Posted

Claims

Findings

Headline finding

MOMENTKV, which maintains compact moment statistics over the evicted token set to jointly guide eviction and provide a closed-form first-order approximation of the evicted attention output, outperforms all tested KV cache eviction baselines on LongBench and RULER with LLaMA-3.1-8B-Instruct and Qwen3-4B-Instruct at every cache budget, with the largest gains under aggressive compression.

A primary source of output degradation in KV cache eviction is not the residual attention mass on evicted tokens, which existing methods already minimize, but a directional mismatch (near-orthogonality) between the retained and evicted token sub-outputs.

Under H2O selection on LLaMA-3-8B at N=4096 and L=128, the evicted attention mass averages below 0.08, while the retained and evicted sub-outputs remain near-orthogonal across all layers, amplifying residual mass into over 20% relative error on average (with peaks above 60% in shallow layers).

The closed-form first-order approximation of the evicted attention output f^E(q)=vˉ+1nedS~q\hat{f}_{\mathcal{E}}(\mathbf{q}) = \bar{\mathbf{v}} + \frac{1}{n_e \sqrt{d}} \tilde{\mathbf{S}} \mathbf{q} has an approximation error scaling as O(σ2)O(\sigma^2) where σ=max⁡i∈E∣δi∣\sigma = \max_{i \in \mathcal{E}} |\delta_i|, and requires O(d2)O(d^2) total storage per head independent of context length.

Moment-informed eviction (MI) and normalization-corrected inference (NC) form a reinforcing loop that yields super-additive gains when combined, with NC contributing roughly 3 to 4 times more than MI alone.

MOMENTKV's first-order approximation adds O(d^2) storage per head (4.1 MB total, 6.4% of retained cache at L=128 on LLaMA-3.1-8B) and 1.6 ms per-token latency over SnapKV, while reducing per-token latency by over 65% compared to the full cache.

Half-precision (FP16) accumulation of moment statistics introduces negligible numerical error compared to FP32, with an average LongBench score difference of less than 0.05 across all task categories on LLaMA-3.1-8B at L=128.

Methods

Decompose the full attention output into a convex combination of retained and evicted sub-outputs, f(q) = w_R f_R(q) + w_E f_E(q), and factorize the eviction error into a scalar mass term w_E and a vector-valued divergence f_E(q) - f_R(q).

Source excerpts

“Since Z = Z_R + Z_E, the full output decomposes exactly as a convex combination: f(q) = w_R f_R(q) + w_E f_E(q), w_R = Z_R / Z, w_E = Z_E / Z” · Show in paper

Approximate the evicted attention output via a centered softmax first-order Taylor expansion, \hat{f}_E(q) = \bar{v} + (1 / (n_e \sqrt{d})) \tilde{S} q, maintained incrementally using four running sums (n_e, s_k, s_v, S).

Source excerpts

“Applying the first-order Taylor expansion exp(\delta_i) \approx 1 + \delta_i, the zero-sum property \sum_{i\in\mathcal{E}} \delta_i = 0 collapses the denominator to n_e. In the numerator, the constant term produces the mean evicted value \bar{v} = n_e−1e^{-1} \sum_{i\in\mathcal{E}} v_i, and the linear term produces a matrix-vector product involving the empirical value-key covariance \tilde{S} = \sum_{i\in\mathcal{E}} v_i (k_i - \bar{k})^\top \in \mathbb{R}^{d\times d}. Dividing by n_e yields the closed-form approximation \hat{f}_{\mathcal{E}}(q) = \bar{v} + \frac{1}{n_e \sqrt{d}} \tilde{S} q.” · Show in paper

Score retained tokens for eviction using the product of attention weight and moment residual norm, score(j) = \alpha_j \cdot ||r_j||, where r_j = v_j - \bar{v} - (1 / (n_e \sqrt{d})) \tilde{S} k_j, evicting the lowest-scoring token.

Source excerpts

“The eviction score combines the attention weight with the residual norm as score(j) = \alpha_j \cdot ||r_j||, and the token with the lowest score is evicted first.” · Show in paper

Correct the post-eviction attention output using a Jensen lower bound on the evicted partition function, \hat{Z}_E = n_e \exp(q^\top \bar{k} / \sqrt{d}), and compute the mixing weight in the log domain to avoid overflow.

Source excerpts

“This yields the lower bound \hat{Z}_{\mathcal{E}} = n_e \cdot \exp(q^\top \bar{k}/\sqrt{d}) \le Z_{\mathcal{E}}, with relative error O(\sigma^2) controlled by the logit variance (see Appendix B). With both components estimated, the corrected output takes the form \hat{f}(q) = \hat{w}_{\mathcal{R}} f_{\mathcal{R}}(q) + (1 - \hat{w}_{\mathcal{R}}) \hat{f}_{\mathcal{E}}(q), \hat{w}_{\mathcal{R}} = Z_{\mathcal{R}} / (Z_{\mathcal{R}} + \hat{Z}_{\mathcal{E}}).” · Show in paper

“We avoid overflow by computing the mixing weight in the log domain: \log \hat{w}_{\mathcal{R}} = \log Z_{\mathcal{R}} - \log(Z_{\mathcal{R}} + \hat{Z}_{\mathcal{E}}) = \log Z_{\mathcal{R}} - \text{logsumexp}(\log Z_{\mathcal{R}}, \log n_e + \bar{s})” · Show in paper

Clamp entries of the recovered covariance matrix \tilde{S} with magnitude below \tau = 10^{-6} to zero to prevent catastrophic cancellation noise in half-precision.

Source excerpts

“Second, we apply a simple numerical safeguard: after computing \tilde{S}, we clamp any entry whose magnitude falls below a threshold \tau = 10^{-6} to zero, preventing noise in the low-significance bits from propagating through the matrix-vector product \tilde{S} q.” · Show in paper