Search publications
Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving
PaperCodeSource excerpt
“All-KV was not the consistent winner in terms of accuracy across the various workloads in our benchmark. Instead, under most workloads, one of the compression techniques achieved even better results.” · Show in paper
Verified
Table 2 directly shows compressed methods outperforming All-KV on most model-dataset combinations.
Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving
PaperCodeSource excerpt
“The results show that the compression ratio alone is a poor predictor of end-to-end performance.” · Show in paper
Verified
The joint evaluation across task quality, TTFT, throughput, and realized compression ratio clearly demonstrates that compression ratio alone does not predict end-to-end system or quality outcomes.
KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache· Method
PaperCodeSource excerpt
“The B-bit integer quantization-dequantization process can be expressed as: Q(X) = ⌊(X - z_X)/s_X⌉, X' = Q(X) · s_X + z_X, where z_X = min X is the zero-point, s_X = (max X - min X)/(2^B - 1) is the scaling factor, and ⌊·⌉ is the rounding operation.” · Show in paper
Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving· Method
PaperCodeSource excerpt
“For KIVI [33], we use the official KIVI-style asymmetric KV-cache quantization implementation integrated into the Llama and Mistral attention modules. We evaluate both 2-bit and 4-bit KV-cache variants.” · Show in paper
Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving
PaperCodeSource excerpt
“Overall, these results indicate that KV cache compression techniques introduce only limited additional latency for the first token, with the exception of TurboQuant, whose more complex quantization pipeline leads to noticeable overhead.” · Show in paper
Verified
Table 3 and Figure 2 support the claim for the evaluated datasets, though GovReport results for TurboQuant are omitted due to 3-day single-run execution times.
A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation
PaperCodeSource excerpt
“Quality is essentially unchanged in the free zone. On Mistral at T=1024, the reconstruction-error gap between FlashJoLT and JoLT is |∆K| ≤ 0.0093 and |∆V | ≤ 0.0011 across 2–3× (Appendix A). Timing both compressors over the full compression pipeline, including layer grouping, gives end-to-end speedups of 1.04× to 3.97× depending on cache length and target ratio (Appendix A).” · Show in paper
Verified
Appendix A provides both the reconstruction-error parity measurements and the end-to-end wall-clock timing on A100.
TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate
PaperSource excerpt
“Experimental results validate our theoretical findings, showing that for KV cache quantization, we achieve absolute quality neutrality with 3.5 bits per channel and marginal quality degradation with 2.5 bits per channel.” · Show in paper
Verified
The experimental results in Figure 4 and Table 1 directly support the claimed quality neutrality at 3.5 bits and marginal degradation at 2.5 bits.
End-to-End Context Compression at Scale
PaperCodeSource excerpt
“In Figure 13, LCLMs achieve the highest accuracy across all compression ratios on GSM8K, with particularly strong gains over baselines at higher compression ratios.” · Show in paper
Verified
Table 9 and Figure 13 directly support the claim across all three compression ratios.
Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction
PaperCodeSource excerpt
“Across diverse long-context language and vision-language reasoning, and multi-turn dialogue benchmarks, our method substantially reduces KV memory while matching or surpassing full-cache inference. These results suggest that learned, globally calibrated KV eviction is not only a compression technique, but also a mechanism for improving long-context reasoning.” · Show in paper
Verified
The experimental results across Tables 1, 2, 3, 7 and Figure 5 directly support that DBTrimKV reduces KV memory while matching or surpassing full-cache inference across all tested benchmark categories.
RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
PaperCodeSource excerpt
“We show that RocketKV provides a compression ratio of up to 400×, end-to-end speedup of up to 3.7× as well as peak memory reduction of up to 32.6% in the decode phase on an NVIDIA A100 GPU compared to the full KV cache baseline, while achieving negligible accuracy loss on a variety of long-context tasks.” · Show in paper
Verified
The reported 400x compression ratio, 3.7x A100 speedup, 32.6% peak memory reduction, and near-Full-KV accuracy are directly supported by the experimental results in Section 4.2, Section 4.3, Figure 5, Figure 7, and Table 2.
A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation
PaperCodeSource excerpt
“Across five models from four architecture families, covering multi-head attention, grouped-query attention, and mixture-of-experts architecture, JoLT achieves 2–3× compression with less than 0.2% perplexity degradation, without retraining. On RULER at 64K context with LLaMA-3.1-8B, retrieval accuracy remains near-lossless through 3× and declines by only 0.90 and 2.40 pp at 4× and 5×, respectively.” · Show in paper
Verified
The experimental results in Table 1, Tables 4-7, and Table 9 support the 2-3x near-lossless claim across the tested models and tasks, noting that LLaMA-3.1-8B and Qwen2.5-14B used FlashJoLT.
TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate· Method
PaperSource excerpt
“These non-integer bit precisions result from our strategy of splitting channels into outlier and non-outlier sets, and applying two independent instances of TurboQuant to each, allocating higher bit precision to outliers.” · Show in paper
SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference
PaperCodeSource excerpt
“Across five backbones and four benchmarks at a 10% GPU-resident KV budget, SEKV outperforms every method that fixes resolution at prefill, improving over the strongest semantic baseline by 2.9 points (5.5% relative).” · Show in paper
Verified
The averages in Table 1 and per-backbone breakdown in Table 7 directly support the 2.9-point (5.5% relative) improvement over SentenceKV and the superiority over all tested fixed-resolution baselines.
RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression· Method
PaperCodeSource excerpt
“As shown in Figure 3, RocketKV performs coarse-grain KV cache eviction in the first stage. The purpose of this stage is to remove KV tokens with low importance while keeping the majority of important tokens. In the second stage, it conducts fine-grain dynamic KV token selection on the remaining KV tokens, followed by top-k sparse attention.” · Show in paper
KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache· Method
PaperCodeSource excerpt
“our key idea to solve this problem is to group key cache every G tokens and quantize them separately. Because the number of tokens in X_K can be arbitrary, we split X_K into two parts, namely, the grouped part = X_K[: l - r] and residual part = X_K[l - r :], where l is the number of tokens inside the current key cache X_K, r is the number of residual tokens, where l - r can be divisible by G.” · Show in paper
MomentKV: Closing the Directional Gap in KV Cache Eviction for Long-Context Inference
PaperSource excerpt
“The first-order approximation adds O(d^2) storage per head (4.1 MB total, 6.4% of retained cache) and 1.6 ms latency, yet more than triples the zeroth-order gain by capturing query-dependent directional variation. Both costs are determined by d and do not grow with context length, making MOMENTKV practical for arbitrarily long sequences while reducing per-token latency by over 65% compared to the full cache.” · Show in paper
Verified
Table 4 confirms the 1.6 ms overhead over SnapKV and >65% latency reduction vs. Full Cache (9.8 vs 28.5 ms). Note that Table 5 lists 8.5 MB (6.4%) for LLaMA-3.1-8B whereas the text on page 9 says 4.1 MB (6.4%), which is a minor internal discrepancy in the reported MB figure.
MomentKV: Closing the Directional Gap in KV Cache Eviction for Long-Context Inference
PaperSource excerpt
“On LongBench and RULER with LLaMA-3.1-8B-Instruct and Qwen3-4B-Instruct, MOMENTKV outperforms all baselines at every cache budget, with the largest gains under aggressive compression.” · Show in paper
Verified
The experimental results in Tables 1 and 2 directly support the claim across all tested models, benchmarks, and cache budgets.
KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
PaperCodeSource excerpt
“From this analysis, we developed a tuning-free 2bit KV cache quantization algorithm named KIVI. With hardware-friendly implementation, KIVI can enable Llama, Falcon, and Mistral models to maintain almost the same quality while using 2.6× less peak memory (including model weight). This reduction in memory usage enables up to 4× larger batch size, bringing 2.35× ∼ 3.47× throughput on real LLM inference workload.” · Show in paper
Unsupported
The headline claim overgeneralizes: Section 4.2.2 explicitly notes that Falcon-7B needs 4bit KIVI to maintain accuracy and suffers a large accuracy drop with 2bit KIVI, and the 2.6x memory reduction and throughput gains were only measured on Llama-2-7B.
FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving· Method
PaperCodeSource excerpt
“Our composable format design allows for the decomposition of the KV cache sparse matrix based on prior knowledge. For instance, if certain requests share a prefix, the corresponding rows and columns in the KV cache form a dense submatrix. We can then use a block sparse matrix with a larger B_r to store these submatrices efficiently.” · Show in paper
Gemma 4 Technical Report
PaperSource excerpt
“Combined with KV cache sharing (Shazeer, 2019) and the reuse of keys as values in global layers (Kayyam et al., 2026), these optimizations reduce the global KV cache footprint by up to 37.5%.” · Show in paper
Verified
The reduction follows directly from the specified architectural parameterization ( -RoPE and key-value reuse) and is consistent with the KV cache footprints reported in Table 3.