Search publications
Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction
PaperCodeSource excerpt
“Theorem 3.3 (Geometric decay of retention (Informal)). Assume that, within an attention head, future query states evolve according to stable dynamics. If, whenever the query state is inside token 's relaxed top- region , there is probability at least of exiting this region within the next decoding steps, then there exists such that .” · Show in paper
Verified
The mathematical derivation in Appendix A.2 is sound under Assumptions A.1 and A.2, and Figures 2, 8, and 9 support the empirical plausibility of the assumptions and decay.
RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
PaperCodeSource excerpt
“Presented in the second row of Figure 5, RocketKV achieves near the Full-KV accuracy across all models, even with a token budget of 256. In fact, it reaches 100% accuracy on Llama3.1-8B-Ins under 256 token budget, which corresponds to a compression ratio of over 400× with a maximum sequence length of 109K tokens. In contrast, all other methods suffer from substantial accuracy drops.” · Show in paper
Verified
The NIAH scores in Figure 5(d-f), Table 2, Table 10, and Table 16 directly support the claim.
End-to-End Context Compression at Scale
PaperCodeSource excerpt
“Online latent compaction is nearly lossless, reaching 90.52% accuracy compared with 92.46% for full-context generation, while being highly efficient: it reduces the average KV length by 3.1x, the average per-problem peak by 3.6x, and the maximum peak by 7.9x: the longest full-context trajectories reach the 32,768-token limit, whereas no compacted trajectory exceeds 4,135 tokens.” · Show in paper
Verified
The full 5,000-problem MATH test set evaluation under matched prompts, context limits, and scoring directly supports the accuracy and KV-reduction claims.
Bursts That Outlast Themselves: Persistent Degradation in Prefix-Cached Agent Serving
PaperSource excerpt
“Once collapsed, throttling admissions recovered 2 of 8 runs after 21–23 minutes and latency-driven AIMD (which also suspends sessions) 5 of 7 after 14–25 minutes, while suspending admitted sessions at turn boundaries against a limit derived from the KV pool restored throughput and a 0.93–0.95 hit ratio in every collapsed run within 6 minutes.” · Show in paper
Verified
Matched-seed recovery experiments and Table 3 directly support the superiority of capacity-based session parking over admission-only throttling.
Efficient Memory Management for Large Language Model Serving with PagedAttention· Method
PaperCodeSource excerpt
“The KV block manager also maintains block tables—the mapping between logical and physical KV blocks of each request. Each block table entry records the corresponding physical blocks of a logical block and the number of filled positions. Separating logical and physical KV blocks allows vLLM to dynamically grow the KV cache memory without reserving it for all positions in advance” · Show in paper
SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference
PaperCodeSource excerpt
“while transferring 3× less data per decoding step than ShadowKV and using the least total memory among the methods we profile. At 128K context, it halves peak GPU memory and cuts total memory by one third relative to full KV caching.” · Show in paper
Verified
Tables 2 and 20 directly confirm the 3.1x transfer reduction vs. ShadowKV, lowest total memory (46.01 GB), ~50% peak GPU memory reduction, and ~34% total memory reduction vs. Full KV.
Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction
PaperCodeSource excerpt
“DBTrimKV adds a small overhead compared to TrimKV due to variable-length, head-specific cache handling; however, it still outperforms vanilla inference at scale.” · Show in paper
Verified
Figures 7, 10, 11, and 12 demonstrate that DBTrimKV is substantially faster than Vanilla at scale while incurring a modest overhead relative to TrimKV.
FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving· Method
PaperCodeSource excerpt
“Query and output matrices are efficiently stored as ragged tensors (also known as jagged arrays) (Tensorflow Developers, 2018) without padding, which facilitates the compact packing of queries and outputs from diverse requests into a single tensor.” · Show in paper
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“Inspect at most H2 + O(1) bits of the accuracy input. If its complete encoding has not been obtained, use exact alignment dynamic programming. The unread suffix is neither copied nor addressed.” · Show in paper
Claim A CPU KV offload tier delays the collapse but does not prevent it.
Bursts That Outlast Themselves: Persistent Degradation in Prefix-Cached Agent Serving
PaperSource excerpt
“A CPU offload tier delays the collapse but does not prevent it.” · Show in paper
Verified
The paired CPU-offload experiments directly demonstrate delayed but unprevented persistent degradation.
BridgeKV: Large-Model Prefill and Small-Model Decoding for LLM Inference via KV Cache Translation
PaperSource excerpt
“For every plotted setting, BridgeKV improves over the fine-tuned decode anchor while keeping the same small model for generation. However, the magnitude of the gain depends on the task and model pair. As shown, BridgeKV improves the accuracy of the directly fine-tuned decode anchor by up to 8.4% relative (NarrativeQA, for Qwen2.5-Instruct-7B→Qwen2.5-Instruct-0.5B). In the worst case, the relative improvement is 0.2% (SAMSum, for Llama-3.1-Instruct-8B→Llama-3.2-Instruct-3B)” · Show in paper
Verified
Figure 3 confirms that BridgeKV outperforms the parameter-matched decode anchor in all 30 settings, with relative gains ranging from 0.2% (SAMSum, Llama 8B->3B: 45.3 to 45.4) to 8.4% (NarrativeQA, Qwen 7B->0.5B: 56.8 to 61.5).
SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference
PaperCodeSource excerpt
“It matches exact-recall offloading caches on LongBench and InfiniteBench and trails on synthetic exact retrieval” · Show in paper
Verified
Table 1 supports the claim: SEKV is slightly ahead on LongBench (+0.54) and nearly level on InfiniteBench (-0.25), while trailing on RULER (-3.32) and NIAH (-4.28).
PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization· Method
PaperSource excerpt
“For each action, the planner estimates accuracy impact by combining three factors: offline global sensitivity, online routing statistics, and runtime prompt residual.” · Show in paper
BridgeKV: Large-Model Prefill and Small-Model Decoding for LLM Inference via KV Cache Translation· Method
PaperSource excerpt
“Unless otherwise stated, the main system results use a 200-request trace with 1024 input tokens and 1024 output tokens per request. Request arrivals follow an exponential distribution with an arrival rate of 2.0 requests/s.” · Show in paper
A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation
PaperCodeSource excerpt
“Dropping the residual entirely (rank-only compression) raises 2× perplexity from 6.54 to 7.27, a gap that additional rank cannot close.” · Show in paper
Verified
Table 11 directly supports both the necessity of the rotated residual and the saturation at 4 bits.
Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction
PaperCodeSource excerpt
“Table 3 shows that DBTrimKV and TrimKV outperform heuristic eviction baselines by a wide margin, with the advantage widening as the KV budget shrinks. Notably, DBTrimKV at all budgets outperforms vanilla inference with full cache (by 14% at M = 512).” · Show in paper
Verified
Table 3 (and extended metrics in Table 6 on page 29) directly supports the claim.
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“Assign independent uniform priorities in [0, 1] to the algorithms and run them in priority order, caching previously read bits.” · Show in paper
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“The optimization below uses finite Gibbs variational duality: subtracting linear constraints from an entropy functional produces a normalized exponential law.” · Show in paper
SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference· Method
PaperCodeSource excerpt
“For each span and head, we factorize the pre-RoPE key matrix and the value matrix with a truncated SVD up to rank , and a lightweight predictor assigns each component a gate from the singular spectrum and the span summary: ” · Show in paper
A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation
PaperCodeSource excerpt
“On Mistral-7B-v0.3 at context length 8,192, peak decode memory of the KV representation is 1.43×, 2.40×, 3.61×, and 4.73× below the dense bf16 cache at the 2×, 3×, 5×, and 8× compression targets.” · Show in paper
Verified
The memory savings and latency overhead are both transparently reported in Section 4.5.