Search publications
KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
PaperCodeSource excerpt
“For key cache, there are a few fixed channels whose magnitudes are very large, which is consistent with previous finding (Lin et al., 2023; Xiao et al., 2023a). Thus, as shown in Figure 1 right, key cache should be quantized per-channel, i.e., group elements along the channel dimension and quantize them together. In this way, it can confine the error to each individual channel, without impacting the other normal channels.” · Show in paper
Verified
Both the visualization in Figure 2 and the quantitative error metrics in Table 2 directly support that key cache has channel-wise outliers and suffers much lower error under per-channel quantization.
Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving
PaperCodeSource excerpt
“SnapKV achieves the best throughput, on par with All-KV on Llama and even surpassing All-KV on Mistral.” · Show in paper
Verified
Figure 3 directly supports that SnapKV-0.75 achieves the highest average throughput among all evaluated methods on both models.
MomentKV: Closing the Directional Gap in KV Cache Eviction for Long-Context Inference
PaperSource excerpt
“We identify that a primary source of output degradation is not the residual attention mass on evicted tokens, which existing methods already minimize, but a directional mismatch between the retained and evicted token sets. Specifically, the evicted tokens in practice are often near-orthogonal to the retained ones. Thus, even a small evicted mass could have an oversized impact on the resulting direction distribution and amplify into substantial output error.” · Show in paper
Verified
The mathematical decomposition in Section 3 and empirical measurements in Figure 2 substantiate that directional mismatch drives the output error.
FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving
PaperCodeSource excerpt
“For decode kernels, the performance gap between sparse and dense KV-Cache is negligible (within 1%). For prefill kernels, there is approximately a 10% performance gap.” · Show in paper
Verified
Figure 12 shows decode bandwidth utilization within 1% (83–85%) and prefill TFLOPs/s within ~5–15% (around 10% on average).
Resource-Efficient Speculative Decoding for Long-Context LLM Serving· Method
PaperSource excerpt
“SpecStream uses online softmax to maintain normalization state chunk by chunk [17, 18]. For each query and attention head, it stores three accumulators (m, ℓ,a).” · Show in paper
SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference
PaperCodeSource excerpt
“Storage: spending the same CPU memory on exact KV for a subset of spans costs 2.9 to 3.5 points, because low-rank factors cover every span, and a fixed rank of 32 matches the learned rank in accuracy but needs 1.6× the CPU memory.” · Show in paper
Verified
Table 3 directly supports both the 2.9–3.5 point drop for exact KV at matched CPU bytes and the ~1.6x CPU memory increase for fixed rank 32.
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“The fixed-pair estimate below combines polynomial evaluation, as in classical algebraic fingerprinting [32], with affine universal hashing [11].” · Show in paper
BridgeKV: Large-Model Prefill and Small-Model Decoding for LLM Inference via KV Cache Translation· Method
PaperSource excerpt
“Gψ,j(z) = Pj z̄ + bj + αj Dj σ(Uj z̄), z̄ = LN(z)” · Show in paper
Efficient Memory Management for Large Language Model Serving with PagedAttention
PaperCodeSource excerpt
“Indeed, our profiling results in Fig. 2 show that only 20.4% - 38.2% of the KV cache memory is used to store the actual token states in the existing systems.” · Show in paper
Verified
The profiling data in Figure 2 directly supports the 96.3% vs 20.4%-38.2% memory utilization comparison.
Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving
PaperCodeSource excerpt
“CaM yields large gains on selected QA workloads but exhibits substantial workload sensitivity in both quality and realized compression ratio.” · Show in paper
Verified
Both Table 2 and Table 4 clearly demonstrate extreme workload sensitivity in CaM's accuracy and realized compression ratio.
Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving· Method
PaperCodeSource excerpt
“For the question answering tasks HotpotQA, 2WikiMQA, Qasper, and MultiFieldQA_en, we use the F1 score, which measures the overlap between predicted and ground-truth answers. For TriviaQA, we use the exact match (EM)” · Show in paper
End-to-End Context Compression at Scale
PaperCodeSource excerpt
“LCLMs achieve the fastest compression time at every context length and substantially lower peak memory at long contexts; all baselines run out of memory at 512K tokens, except Attention Matching, which fails at 512K due to numerical instability in its linear solver and runs out of memory at 1M.” · Show in paper
Verified
Figure 4 and Section 6.2 clearly document the scaling measurements and memory limits on the H200 GPU.
KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
PaperCodeSource excerpt
“As shown in Table 3, we observe that for the Llama and Mistral model, KIVI only has up to 2% accuracy drop despite the KV cache being stored in 2bit. ... Since Falcon-7B adopts multi-query attention and only has one head for KV cache, it is already highly compressed compared to Llama-based models. Thus, in Table 3, 4bit KIVI is needed to maintain the accuracy, while 2bit KIVI may have a large accuracy drop in this case.” · Show in paper
Verified
The reported tables and figures accurately reflect these accuracy bounds and the distinction between Llama/Mistral and Falcon-7B.
End-to-End Context Compression at Scale
PaperCodeSource excerpt
“We introduce Latent Context Language Models (LCLMs), a family of compressors that improve the Pareto frontier across general-task performance, compression speed, and peak memory usage.” · Show in paper
Verified
The accuracy, TTFT, and peak memory benchmarks across RULER, LongBench, LongHealth5, and GSM8K support the claim that LCLMs improve the Pareto frontier over KV cache compression baselines.
KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
PaperCodeSource excerpt
“The difference between fake “2bit (K per-channel, V per-token)” quantization and KIVI is that KIVI maintains a full precision key and value cache sliding window for the local relevant tokens. This sliding window is crucial to maintaining accuracy for hard generation tasks such as mathematical reasoning.” · Show in paper
Verified
Table 3 clearly shows that fake 2bit (K-C, V-T) degrades sharply on GSM8K, while KIVI-2 recovers most of the 16bit baseline performance.
KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
PaperCodeSource excerpt
“For value cache, there is no obvious outlier pattern. Although value cache has no obvious outlier pattern, we experimentally show that it can only be quantized per-token because it is used to calculate the attention output, which is essentially a value cache mixer. As shown in Figure 1 left, the per-token quantization can confine the error inside each individual token and ensure that the quantization of one token does not adversely impact the others.” · Show in paper
Verified
The sparsity measurement, the 15x reduction in relative attention output error in Table 2, and the downstream task results in Tables 1 and 3 strongly support this claim.
Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction
PaperCodeSource excerpt
“Notably, DBTrimKV with budgets of 1024, 512, and even 256 surpasses vanilla full-cache inference with up to 3.75%.” · Show in paper
Verified
Table 2 (103.73% at budget 512, i.e., +3.73%, and 103.26% at 1024) and Figure 5 support the claim.
Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction· Method
PaperCodeSource excerpt
“At inference time, we use the learned retention coefficients to rank all cached KV entries globally. For each cached token (ℓ, h, i) present at decoding step t, we assign the retention score G̃ℓ,h,i(t) = ∑T s=t+1 βs−i ℓ,h,i = βt+1−i ℓ,h,i (1 − βT−t ℓ,h,i)/(1 − βℓ,h,i), for βℓ,h,i < 1.” · Show in paper
Resource-Efficient Speculative Decoding for Long-Context LLM Serving· Method
PaperSource excerpt
“The Target KV Cache is partitioned into the following three regions accordingly, Sealed History: [0,H) (CPU) Active Tail: [H,Cr) (GPU) Speculative Frontier:[Cr ,Lr)(GPU) 0 ≤ H ≤ Cr ≤ Lr.” · Show in paper
Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction
PaperCodeSource excerpt
“As shown in Table 1, both DBTrimKV and TrimKV consistently outperform visual token pruning baselines and match vanilla performance under all memory budgets.” · Show in paper
Verified
Table 1 directly supports the claim, showing 99.4%–99.9% relative performance compared to vanilla and higher scores than all visual token pruning baselines.