By Nikita Agrawal, Ruben MayerSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2607-05399.git
Cite
Version link
https://openversions.com/index/arxiv-2607-05399/commit/3e258b66da79621625f80a9a12f2da1d84294a06
Version identifier (commit hash)
3e258b66da79621625f80a9a12f2da1d84294a06

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Latest version

Checks complete

Archive from arXiv Source: https://arxiv.org/abs/2607.05399v1 OpenVersions: b0e19d27-b301-4942-85d0-dcfa197a51ab

Posted

Claims

Findings

Headline finding

Compression ratio alone is a poor predictor of end-to-end performance for KV-cache compression methods, motivating workload-aware selection rather than one-size-fits-all compression.

Moderate quantization with KIVI4 provides the most stable task quality across workloads and models (Llama-3.1-8B-Instruct and Mistral-7B-Instruct-v0.3).

SnapKV delivers the strongest long-context output throughput among the evaluated KV-cache compression methods, performing on par with All-KV on Llama-3.1-8B-Instruct and surpassing All-KV on Mistral-7B-Instruct-v0.3.

CaM yields large accuracy gains on selected QA workloads but exhibits substantial workload sensitivity in both task quality and realized compression ratio.

KV-cache compression techniques introduce only limited additional time-to-first-token (TTFT) latency compared to All-KV, with the exception of TurboQuant, which exhibits the highest TTFT overhead.

Lossy KV-cache compression does not necessarily trade accuracy for memory efficiency; under most workloads in the benchmark, at least one compression technique achieved higher accuracy than the All-KV baseline.

Aggressive rotation-based quantization in TurboQuant leads to substantial quality drops on summarization and noticeable degradation (~7%) on few-shot learning with Llama-3.1-8B-Instruct, as well as severe throughput degradation.

Methods

Task quality is evaluated on six LongBench datasets spanning four categories (HotpotQA and 2WikiMQA for multi-document QA; Qasper and MultiFieldQA_en for single-document QA; TriviaQA for few-shot learning; MultiNews for summarization), and system efficiency is evaluated on NarrativeQA, Qasper, and GovReport across context-length buckets.

Source excerpts

“For task quality evaluation, we use six representative datasets: HotpotQA and 2WikiMQA for multi-document QA, Qasper and MultiFieldQA_en for single-document QA, TriviaQA for few-shot learning, and MultiNews for summarization.” · Show in paper

“For system efficiency evaluation, we focus on NarrativeQA, Qasper, and GovReport, as they contain substantially longer contexts, making them better suited to capturing realistic KV cache behavior and more representative system-level performance under long-context workloads.” · Show in paper

Experiments are conducted on Llama-3.1-8B-Instruct and Mistral-7B-Instruct-v0.3 loaded in FP16 with FlashAttention2 on an NVIDIA A100 GPU with 40GB of memory.

Source excerpts

“We conduct experiments on two widely used instruction-tuned models: Llama-3.1-8B-Instruct [16] and Mistral-7B-Instruct-v0.3 [21].” · Show in paper

“Models were loaded in FP16 with FlashAttention2 [9] enabled, and prompts were formatted using the corresponding chat template for each instruction-tuned model.” · Show in paper

“We conduct our experiments on a NVIDIA A100 GPU with 40GB of memory.” · Show in paper

Four representative training-free KV-cache compression methods are benchmarked: KIVI (2-bit and 4-bit, group size 32, residual length 32, with FP16 as All-KV baseline), TurboQuant (3-bit outlier-aware with 32/128 outlier channels at 4 bits and 4-bit uniform, unpacked cache), SnapKV (compression ratio 0.75, window size 32, kernel size 7), and CaM (CAMPress with SnapKV base, compression interval 32, target cache size 1024, hidden-state buffer 64, merge budget 32, window size 16, kernel size 7).

Source excerpts

“For KIVI [33], we use the official KIVI-style asymmetric KV-cache quantization implementation integrated into the Llama and Mistral attention modules. We evaluate both 2-bit and 4-bit KV-cache variants.” · Show in paper

“For TurboQuant [46], we used the Transformers 4.45-compatible TurboQuant cache implementation by Omar Hory [19]. We evaluated 3-bit and 4-bit KV-cache quantization.” · Show in paper

“For SnapKV, we used the SnapKVPress implementation from KVPress [10]. The cache was compressed just after prefill using SnapKV's attention-based token selection [29]. We set the compression ratio to 0.75.” · Show in paper

“For CaM [48], we evaluated the CAMPress wrapper with SnapKV as the base compressor provided by KVPress [10]. We used CAMPress with a compression interval of 32” · Show in paper

Evaluation metrics comprise LongBench task accuracy metrics (F1 for QA, Exact Match for TriviaQA, ROUGE for MultiNews) and system efficiency metrics (time-to-first-token, output throughput in tokens/sec, and prefill KV cache memory / compression rate).

Source excerpts

“For the question answering tasks HotpotQA, 2WikiMQA, Qasper, and MultiFieldQA_en, we use the F1 score, which measures the overlap between predicted and ground-truth answers. For TriviaQA, we use the exact match (EM)” · Show in paper

“We measure efficiency metrics such as time-to-first-token (TTFT), output throughput, and prefill KV memory.” · Show in paper