By Ngoc Bui, Hieu Trung Nguyen, Arman Cohan, Rex YingSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2605-09649.git
Cite
Version link
https://openversions.com/index/arxiv-2605-09649/commit/37478af3a581cd4dc579ed9e4fdfa0010bce8d62
Version identifier (commit hash)
37478af3a581cd4dc579ed9e4fdfa0010bce8d62

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Latest version

Checks complete

Archive from arXiv Source: https://arxiv.org/abs/2605.09649v1 OpenVersions: dd2049ed-ae4e-4cde-883d-912e0a5c9174

Posted

Claims

Findings

Headline finding

Learned, globally calibrated KV eviction (DBTrimKV) substantially reduces KV memory while matching or surpassing full-cache inference across diverse long-context language, vision-language reasoning, and multi-turn dialogue benchmarks, suggesting it is both a compression technique and a mechanism for improving long-context reasoning.

In long contexts, near-tie distractors force severe attention dilution (δt→1\delta_t \to 1 and ∑i∈Utαt,i→0\sum_{i \in U_t} \alpha_{t,i} \to 0 as ∣Dt′∣→∞|D'_t| \to \infty), and any preferential retention rule that suppresses distractor mass more than useful-token mass (ρD≤ρU\rho_D \le \rho_U) reduces attention dilution (δtr≤δt\delta^r_t \le \delta_t).

Under stable low-rank future query dynamics and a top-K block-exit condition, a cached token's cumulative future utility is upper-bounded by a geometrically decaying function Gˉi(t)≤Ai(t)∑s=t+1Twt,sβis−t\bar{G}_i(t) \le A_i(t) \sum_{s=t+1}^T w_{t,s} \beta_i^{s-t} with βi=(1−ϵi)1/bi∈(0,1)\beta_i = (1-\epsilon_i)^{1/b_i} \in (0, 1), justifying geometric retention rt,i=βit−ir_{t,i} = \beta_i^{t-i} as a query-agnostic proxy for future utility.

On short-form VQA benchmarks (GQA, VQAText, MME) with LLaVA-1.5-7B, both DBTrimKV and TrimKV consistently outperform visual token pruning baselines and match vanilla full-cache performance under all tested memory budgets (128, 64, and 32 visual tokens).

On long-horizon image, video, and pure-text reasoning benchmarks, DBTrimKV achieves state-of-the-art performance compared to KV eviction baselines, with particularly large gains in low-budget regimes, and surpasses vanilla full-cache inference at budgets of 1024, 512, and 256 by up to 3.75%.

On multi-turn visual dialogue (MMDU) with Qwen3-VL-4B-Instruct, DBTrimKV outperforms heuristic eviction baselines and TrimKV by a wide margin, and outperforms vanilla full-cache inference at all tested budgets (512, 256, and 128), including by 14% at M=512M=512.

With PagedAttention, DBTrimKV adds a small runtime overhead compared to TrimKV due to variable-length, head-specific cache handling, but still outperforms vanilla full-cache inference in decoding time and throughput at scale.

Weight tying in the final linear layer of the retention gates yields a substantial performance improvement on MathVision, especially in the low-budget regime.

Using the token contextual embedding xt\mathbf{x}_t as input to the retention gate performs better on MathVision than using the concatenation of key and value vectors [kt∥vt][\mathbf{k}_t \| \mathbf{v}_t], particularly under tight budgets.

Methods

Weight-tied retention gates: Per-layer, per-head retention gates gℓ,h(x)=σ(wg⊤Projℓ,h(x)+bg)g_{\ell,h}(\mathbf{x}) = \sigma(\mathbf{w}_g^\top \text{Proj}_{\ell,h}(\mathbf{x}) + b_g) compute a head-specific embedding via a two-layer MLP Projℓ,h\text{Proj}_{\ell,h} and produce a calibrated scalar retention score βℓ,h,t∈[0,1]\beta_{\ell,h,t} \in [0, 1] using a shared final linear projection (wg,bg)(\mathbf{w}_g, b_g) tied across all layers and heads.

Source excerpts

“Each gate first computes a head-specific embedding, but the final scalar score is produced by a shared projection: gℓ,h(x) = σ (w⊤g Projℓ,h(x) + bg)” · Show in paper

“where Projℓ,h is layer/head-specific, while (wg, bg) is tied across all layers and heads. This shared readout calibrates retention scores globally: a score produced in one head has the same meaning as the same score produced in another.” · Show in paper

Global capacity constraint training: Retention gate weights are trained with LLM parameters frozen using a quality distillation/cross-entropy loss Lquality\mathcal{L}_{\text{quality}} and a global capacity constraint Lcap=∑t=1Tmax⁡(0,∑ℓ,h∑i=1tβℓ,h,it−i−Mglobal)\mathcal{L}_{\text{cap}} = \sum_{t=1}^T \max(0, \sum_{\ell,h} \sum_{i=1}^t \beta_{\ell,h,i}^{t-i} - M_{\text{global}}).

Source excerpts

“We follow the same training procedure as [Bui et al., 2025] described in Section 2.2 but replace the local capacity constraint by a global counterpart Lcap = ∑T t=1 max(0, ∑ℓ,h ∑t i=1 βt−i ℓ,h,i − Mglobal), where Mglobal is the target global KV budget. We only train retention gates and freeze LLM weights.” · Show in paper

Global KV eviction with lookahead horizon: At inference step tt, each cached token (ℓ,h,i)(\ell, h, i) is assigned a score G~ℓ,h,i(t)=∑s=t+1Tβℓ,h,is−i=βℓ,h,it+1−i1−βℓ,h,iT−t1−βℓ,h,i\tilde{G}_{\ell,h,i}(t) = \sum_{s=t+1}^T \beta_{\ell,h,i}^{s-i} = \beta_{\ell,h,i}^{t+1-i} \frac{1 - \beta_{\ell,h,i}^{T-t}}{1 - \beta_{\ell,h,i}} (with default lookahead horizon T−t=2T-t=2), and the MglobalM_{\text{global}} tokens with the largest scores across all layers and heads are retained.

Source excerpts

“At inference time, we use the learned retention coefficients to rank all cached KV entries globally. For each cached token (ℓ, h, i) present at decoding step t, we assign the retention score G̃ℓ,h,i(t) = ∑T s=t+1 βs−i ℓ,h,i = βt+1−i ℓ,h,i (1 − βT−t ℓ,h,i)/(1 − βℓ,h,i), for βℓ,h,i < 1.” · Show in paper

“Our eviction rule is simple now: retain the Mglobal tokens with the largest scores G̃ℓ,h,i(t) across all layers and heads.” · Show in paper

PagedAttention with head-specific variable-length caches: KV entries are stored in fixed-size pages with per-head block tables, and attention is computed using variable-length kernels to support dynamic per-head sequence lengths without contiguous physical storage.

Source excerpts

“Specifically, KV entries are stored in fixed-size pages, and each head maintains a block table for its currently active pages. Attention is then computed with variable-length kernels [Dao, 2023] using the resulting per-head sequence lengths.” · Show in paper