By Amirhossein Abaskohi, Giuseppe Carenini, Peter West, Yuhang HeSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2606-31145.git
Cite
Version link
https://openversions.com/index/arxiv-2606-31145/commit/c2db6540b985573ec2227baa4af65fa5bc03d7f1
Version identifier (commit hash)
c2db6540b985573ec2227baa4af65fa5bc03d7f1

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Latest version

Checks complete

Archive from arXiv Source: https://arxiv.org/abs/2606.31145v2 OpenVersions: 89a408cd-96d0-47f3-859a-81cee0e51e1b

Posted

Claims

Findings

Headline finding

Across five backbones and four benchmarks at a 10% GPU-resident KV budget, SEKV outperforms every KV cache compression method that fixes resolution at prefill, improving over the strongest semantic baseline by 2.9 points (5.5% relative).

At a 10% GPU-resident KV budget, SEKV matches exact-recall offloading caches on LongBench and InfiniteBench while trailing them on synthetic exact retrieval (NIAH and RULER).

At 128K context, SEKV transfers 3x less data per decoding step than ShadowKV, uses the least total memory among profiled methods, halves peak GPU memory, and cuts total memory by one third relative to full KV caching.

SEKV's trained zoom-in mechanism adds under 0.1% parameters to a frozen base LLM, and replacing it with training-free routing costs 2.8 to 3.8 accuracy points.

Using a single global threshold instead of per-head thresholds in SEKV costs about 2 accuracy points.

Spending the same CPU memory on exact KV for a subset of spans instead of low-rank factors for every span costs 2.9 to 3.5 accuracy points, while a fixed rank of 32 matches the learned rank in accuracy at 1.6x the CPU memory.

Replacing surprisal-based segmentation with fixed-length chunks of the same mean length costs 2.4 to 2.9 points, and dropping anchor tokens costs 3.0 to 3.9 points.

SEKV's accuracy gap to ShadowKV on RULER and NIAH narrows as the GPU-resident budget increases from 5% to 20%, matching ShadowKV on NIAH on Llama-3.1-8B at 20%.

Methods

Surprisal-based segmentation during prefill computes Ht=−log⁡p(xt∣x<t)H_t = -\log p(x_t \mid x_{<t}), marks positions with Ht>μ+ασH_t > \mu + \alpha\sigma as anchor boundaries (merging boundaries closer than LgapL_{\text{gap}}), keeps anchors as exact KV on GPU, and groups tokens between consecutive anchors into spans.

Source excerpts

“we compute Ht=−log⁡p(xt∣x<t)H_t = -\log p(x_t \mid x_{<t}) during prefill and mark positions with Ht>μ+ασH_t > \mu + \alpha\sigma as boundaries, where μ\mu and σ\sigma are the sequence mean and standard deviation, and boundaries closer than LgapL_{\text{gap}} tokens are merged, keeping the highest-surprisal one.” · Show in paper

“Boundary tokens become anchors, kept as exact KV on GPU, and the tokens between consecutive anchors form a span SS.” · Show in paper

Each span maintains two GPU-resident representations per head: a normalized routing summary KˉS=∑t∈SwtWKt∥∑t∈SwtWKt∥2\bar{K}_S = \frac{\sum_{t \in S} w_t W K_t}{\|\sum_{t \in S} w_t W K_t\|_2} using a learned per-head projection W∈Rd′×dhW \in \mathbb{R}^{d' \times d_h} and surprisal weights wt=Ht/∑k∈SHkw_t = H_t / \sum_{k \in S} H_k, and a coarse entry (kˉS,vˉS)=(∑twtKt,∑twtVt)(\bar{k}_S, \bar{v}_S) = (\sum_t w_t K_t, \sum_t w_t V_t).

Source excerpts

“The routing summary projects token keys into a d′d'-dimensional space with a learned per-head matrix W∈Rd′×dhW \in \mathbb{R}^{d' \times d_h} and averages them with surprisal weights wt=Ht/∑k∈SHkw_t = H_t / \sum_{k \in S} H_k: KˉS=∑t∈SwtWKt∥∑t∈SwtWKt∥2\bar{K}_S = \frac{\sum_{t \in S} w_t W K_t}{\|\sum_{t \in S} w_t W K_t\|_2}.” · Show in paper

“The coarse entry is the surprisal-weighted mean key and value in the original space, kˉS=∑twtKt\bar{k}_S = \sum_t w_t K_t and vˉS=∑twtVt\bar{v}_S = \sum_t w_t V_t, which represents the span in attention when it is not expanded.” · Show in paper

For CPU-offloaded storage, pre-RoPE key and value matrices KS,VS∈R∣S∣×dhK_S, V_S \in \mathbb{R}^{|S| \times d_h} of each span are factorized via truncated SVD up to rank Rmax⁡R_{\max}, and a lightweight predictor gϕg_\phi assigns component gates mi∈(0,1)m_i \in (0, 1) to reconstruct X^S=∑i=1Rmax⁡miσiaibi⊤\hat{X}_S = \sum_{i=1}^{R_{\max}} m_i \sigma_i \mathbf{a}_i \mathbf{b}_i^\top.

Source excerpts

“For each span and head, we factorize the pre-RoPE key matrix KS∈R∣S∣×dhK_S \in \mathbb{R}^{|S| \times d_h} and the value matrix VSV_S with a truncated SVD up to rank Rmax⁡R_{\max}, and a lightweight predictor gϕg_\phi assigns each component a gate mi∈(0,1)m_i \in (0, 1) from the singular spectrum and the span summary: X^S=∑i=1Rmax⁡miσiaibi⊤,X∈{KS,VS}\hat{X}_S = \sum_{i=1}^{R_{\max}} m_i \sigma_i \mathbf{a}_i \mathbf{b}_i^\top, X \in \{K_S, V_S\}” · Show in paper

At each decoding step, each head scores every span with α~S=sig((Wqt)⊤KˉSd′+log⁡∣S∣)\tilde{\alpha}_S = \text{sig}\left(\frac{(W q_t)^\top \bar{K}_S}{\sqrt{d'}} + \log |S|\right) and zS=1[α~S>τ]z_S = \mathbf{1}[\tilde{\alpha}_S > \tau], enforces the GPU budget cap by keeping highest-scoring spans, and attends in one softmax over exact anchors/local window, reconstructed expanded spans, and coarse entries with log⁡∣S∣\log |S| added to their scores.

Source excerpts

“At each decoding step, each head scores every span with an independent gate rather than a softmax, since expansion is a per-span decision: α~S=sig((Wqt)⊤KˉSd′+log⁡∣S∣),zS=1[α~S>τ]\tilde{\alpha}_S = \text{sig}\left(\frac{(W q_t)^\top \bar{K}_S}{\sqrt{d'}} + \log |S|\right), z_S = \mathbf{1}[\tilde{\alpha}_S > \tau]” · Show in paper

“Each head then attends once over a single memory: anchors and the local window as exact KV, expanded spans as reconstructed tokens (Eq. 2), and every other span as its coarse entry (kˉS,vˉS)(\bar{k}_S, \bar{v}_S) with log⁡∣S∣\log |S| added to its score so that one entry competes in proportion to the tokens it stands for” · Show in paper

With the base LLM frozen, the projections WW, thresholds τ\tau, and rank predictor gϕg_\phi are trained on generic RedPajama documents using L=Ldistill+λ1Lzoom+λ2Lrecon+λ3Lbudget\mathcal{L} = \mathcal{L}_{\text{distill}} + \lambda_1 \mathcal{L}_{\text{zoom}} + \lambda_2 \mathcal{L}_{\text{recon}} + \lambda_3 \mathcal{L}_{\text{budget}} with a full-KV teacher and a straight-through estimator for zSz_S.

Source excerpts

“The base LLM stays frozen. We train only the projections WW, the thresholds τ\tau, and the rank predictor gϕg_\phi, under 0.1% of the backbone’s parameters, once per backbone on generic text (Section 4.1). The objective combines four terms: L=Ldistill+λ1Lzoom+λ2Lrecon+λ3Lbudget\mathcal{L} = \mathcal{L}_{\text{distill}} + \lambda_1 \mathcal{L}_{\text{zoom}} + \lambda_2 \mathcal{L}_{\text{recon}} + \lambda_3 \mathcal{L}_{\text{budget}}.” · Show in paper

“For each backbone, the router is trained once on long RedPajama documents (Weber et al., 2024) (arXiv, books, and code), which share no data with any evaluation benchmark, for about 3K steps over 0.5B tokens” · Show in paper