By Rahul Krishnan, Volker SchulzSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2607-12550.git
Cite
Version link
https://openversions.com/index/arxiv-2607-12550/commit/57ce427a24f07d21f28078d7e3ff18fb08ca8c96
Version identifier (commit hash)
57ce427a24f07d21f28078d7e3ff18fb08ca8c96

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Latest version

Checks complete

Archive from arXiv Source: https://arxiv.org/abs/2607.12550v4 OpenVersions: 4adac460-4f98-46dc-8c91-45295ca0bc48

Posted

Claims

Findings

Headline finding

JoLT achieves near-lossless KV-cache compression (2-3x compression with less than 0.2% perplexity degradation) without retraining across diverse model architectures, and this free zone carries over to downstream reasoning and long-context retrieval.

At context length n=1024 (T=512), JoLT (and FlashJoLT for LLaMA-3.1-8B and Qwen2.5-14B) keeps perplexity drift below 0.2% at 2x and 3x compression across five models spanning four architecture families (Mistral-7B, LLaMA-2-13B, OLMoE-1B-7B, LLaMA-3.1-8B, Qwen2.5-14B).

In the 2-3x free zone, JoLT outperforms all evaluated low-rank baselines on perplexity and reconstruction error, whereas fixed-rate 8-bit quantizers have no operating point above 2x and dedicated low-bit quantizers (TurboQuant, KIVI) outperform JoLT near 4x.

FlashJoLT matches JoLT within the 2-3x free zone (reconstruction-error gap ∣ΔK∣≤0.0093|\Delta_K| \le 0.0093 and ∣ΔV∣≤0.0011|\Delta_V| \le 0.0011 on Mistral at T=1024) while achieving end-to-end compression pipeline speedups of 1.04x to 3.97x depending on cache length and target ratio.

On RULER at 64K context with LLaMA-3.1-8B, FlashJoLT retrieval accuracy remains near-lossless through 3x (+0.23 pp at 2x, -0.10 pp at 3x) and declines by only 0.90 pp at 4x and 2.40 pp at 5x.

JoLT is near-lossless at 2x and 3x on GSM8K (Mistral-7B and LLaMA-2-13B, within 95% CIs of zero) and FlashJoLT stays within 0.52 pp of the uncompressed pooled score on six LongBench tasks at 16K context on Mistral-7B through 4x.

The fused Triton decode kernel evaluates attention directly over the compressed representation without dense reconstruction, reducing peak KV decode memory by 1.43x to 4.73x across 2-8x targets on Mistral-7B-v0.3 at context length 8,192, though it is 7.5x to 65.9x slower than uncompressed attention at batch 1.

Dropping the rotated residual (rank-only compression) raises 2x perplexity on Mistral-7B-v0.3 from 6.54 to 7.27, a gap that additional rank cannot close, and residual bits saturate at 4 bits (4 to 8 bits improves perplexity by less than 0.001 at 2x).

Per-group rank allocation beats uniform rank at all 15 grid cells (peak advantage 0.97 PPL at n=512, 4x), and joint Lagrangian allocation matches K/V-decoupled allocation at 2x while pulling ahead at 8x (except at n=2048 where they stay within 0.17 PPL).

Methods

Layer grouping: partition L layers into G contiguous groups by computing the top eight token-mode singular vectors of each layer's key cache, scoring consecutive layers by mean squared cosine between bases, and starting a new group when the score falls below the network median (subject to group size bounds of 2 to 8 layers).

Source excerpts

“For each layer, we compute the top eight token-mode singular vectors of its key cache and score consecutive layers by the mean squared cosine between their bases. A new group starts whenever the score falls below the median score across the network.” · Show in paper

Partial Tucker backbone: apply sequentially truncated higher-order SVD (ST-HOSVD) to pre-RoPE key and value fourth-order tensors, truncating only the token and feature modes to ranks r_T and r_d while keeping identity factors on the layer and head modes.

Source excerpts

“The partial Tucker approximation X˜(rT , rd) computes the mode-k singular vectors of the token and feature unfoldings via the sequentially truncated higher-order SVD (ST-HOSVD) [22] and truncates them to ranks rT and rd, while keeping identity factors on the layer and head modes.” · Show in paper

Rotated low-bit residual: apply a seeded random orthogonal rotation (from QR decomposition of a seeded Gaussian) along the feature axis to the truncation residual E = X_g - X_tilde(r_T, r_d), then uniformly quantize to b in {0, 2, 4, 8} bits per element with one fp32 scale per row.

Source excerpts

“We apply a random orthogonal rotation along the feature axis to spread residual energy evenly across channels, so that no single outlier channel dominates the quantizer’s dynamic range. The rotation matrix is the dh × dh factor from the QR decomposition of a seeded Gaussian, shared by every cell and regenerated from its seed at decode, so it costs no storage. We then quantize the rotated entries uniformly at b ∈ {0, 2, 4, 8} bits per element with one fp32 scale per row, where b=0 means no residual is stored.” · Show in paper

Joint Lagrangian allocation: minimize summed modeled reconstruction error eg,te_{g,t}(r_T, r_d, b) = epsilon^2(b) * tau_{g,t}(r_T, r_d) subject to a global byte budget B via Lagrangian relaxation and geometric bisection on the multiplier lambda.

Source excerpts

“Given a total byte budget B for the compressed prefill cache, JoLT minimizes the summed error subject to the summed cost: min {(rT ,rd,b)g,t} X g,t eg,t(rT , rd, b) s.t. X g,t sg,t(rT , rd, b) ≤ B.” · Show in paper

FlashJoLT: replace the exact token-mode SVD with a randomized SVD computing only the leading q_cap = min(max(q_min(C), ceil(T/32)), 512) singular vectors, and append the remaining Frobenius energy as a single synthetic tail entry for the Lagrangian allocator.

Source excerpts

“FlashJoLT replaces this SVD with a randomized approximation [11] that computes only the leading qcap token-mode singular vectors. The feature-mode SVD is kept exact because dh is small (128), and the rest of the pipeline, including the Lagrangian allocator, is unchanged.” · Show in paper

Fused Triton decode kernel: evaluate attention directly over the stored Tucker factors and packed residuals without writing a dense KV buffer to global memory, reconstructing key tiles on-chip with RoPE applied per tile while accumulating value softmax weights in rank space.

Source excerpts

“The FlashJoLT decode kernel evaluates attention directly over the stored factors and packed residual, so no dense KV buffer is ever written to global memory.” · Show in paper