Search publications
ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMs
PaperCodeSource excerpt
“At sequence length 1024, ARCQuant reaches 21077.0 total tokens/s and 2341.9 decode tokens/s, corresponding to a 1.96× decoding speedup over FP16. At sequence length 2048, it achieves 21236.6 total tokens/s and 1249.2 decode tokens/s, yielding a 2.08× decoding speedup over FP16. In both settings, ARCQuant also substantially outperforms the FP8 reference, demonstrating that its efficiency advantages extend beyond prefill and remain effective in decoding generation scenarios.” · Show in paper
Verified
Table 4 shows decode throughput of 2342 vs 1195 (1.96x) at SeqLen 1024 and 1249 vs 600 (2.08x) at SeqLen 2048, outperforming both FP16 and FP8.
RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
PaperCodeSource excerpt
“Again, RocketKV shows robust performance and a clear advantage over all other methods across various token budget and sequence length settings. Overall, we can see that the accuracy loss of RocketKV is negligible under short sequence lengths and gradually becomes larger as sequence length increases.” · Show in paper
Verified
Figure 5(g-r) and the detailed RULER tables (Tables 6-9, 12-15, 18-21) confirm RocketKV's advantage across sequence lengths and token budgets.
MomentKV: Closing the Directional Gap in KV Cache Eviction for Long-Context Inference
PaperSource excerpt
“On LongBench and RULER with LLaMA-3.1-8B-Instruct and Qwen3-4B-Instruct, MOMENTKV outperforms all baselines at every cache budget, with the largest gains under aggressive compression.” · Show in paper
Verified
The experimental results in Tables 1 and 2 directly support the claim across all tested models, benchmarks, and cache budgets.
FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving· Method
PaperCodeSource excerpt
“We developed CUDA/CUTLASS (Thakkar et al., 2023) templates for FlashAttention, designed specifically for both dense and block-sparse matrices and compatible with NVIDIA GPU architectures from Turing to Hopper (sm75 to sm90a).” · Show in paper
BridgeKV: Large-Model Prefill and Small-Model Decoding for LLM Inference via KV Cache Translation· Method
PaperSource excerpt
“All stitching layers of a pair have the same shapes, so the batched implementation stacks their weights and runs the whole module in one pass.” · Show in paper
MomentKV: Closing the Directional Gap in KV Cache Eviction for Long-Context Inference
PaperSource excerpt
“The first-order approximation adds O(d^2) storage per head (4.1 MB total, 6.4% of retained cache) and 1.6 ms latency, yet more than triples the zeroth-order gain by capturing query-dependent directional variation. Both costs are determined by d and do not grow with context length, making MOMENTKV practical for arbitrarily long sequences while reducing per-token latency by over 65% compared to the full cache.” · Show in paper
Verified
Table 4 confirms the 1.6 ms overhead over SnapKV and >65% latency reduction vs. Full Cache (9.8 vs 28.5 ms). Note that Table 5 lists 8.5 MB (6.4%) for LLaMA-3.1-8B whereas the text on page 9 says 4.1 MB (6.4%), which is a minor internal discrepancy in the reported MB figure.
FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving
PaperCodeSource excerpt
“Across all four scenarios and a range of sequence lengths, FlashInfer consistently outperforms FlexAttention, with especially large gains at longer sequence lengths.” · Show in paper
Verified
Tables 1–4 and Tables 9–11 in Appendix G directly substantiate these performance comparisons.
RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
PaperCodeSource excerpt
“Presented in the second row of Figure 5, RocketKV achieves near the Full-KV accuracy across all models, even with a token budget of 256. In fact, it reaches 100% accuracy on Llama3.1-8B-Ins under 256 token budget, which corresponds to a compression ratio of over 400× with a maximum sequence length of 109K tokens. In contrast, all other methods suffer from substantial accuracy drops.” · Show in paper
Verified
The NIAH scores in Figure 5(d-f), Table 2, Table 10, and Table 16 directly support the claim.
SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference
PaperCodeSource excerpt
“The base LLM stays frozen, and the router adds under 0.1% parameters.” · Show in paper
Verified
Table 24 confirms parameter overhead is 0.054%–0.086% (<0.1%), and Table 3 confirms the 2.8–3.8 point drop when using training-free routing.
Bursts That Outlast Themselves: Persistent Degradation in Prefix-Cached Agent Serving
PaperSource excerpt
“The regime appears at loads the cache-warm server sustains, is absent without prefix caching, and also appears with Qwen3-32B on SWE-bench agent trace shapes and on SGLang, which tolerates larger triggers.” · Show in paper
Verified
The SGLang paired runs and 32B trace-shape runs support the generalization, and the authors appropriately note that 32B provides weaker evidence of history dependence than 8B.
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
PaperSource excerpt
“The finetuned models SWE-Llama 7b and 13b perform surprisingly poorly with BM25 retrieved context. As these models were fine-tuned using the “oracle” retrieval as context, we suspect this shift in context makes it difficult for the model to perform reliably. For instance, SWE-Llama was trained to edit every file included as context whereas in the BM25 setting many files provided in context are not expected to be changed.” · Show in paper
Verified
The performance drop from 3.97% (Oracle) to 0.70% (BM25) for SWE-Llama 13b is documented in Tables 5 and 18, and the explanation aligns with the training setup.
BridgeKV: Large-Model Prefill and Small-Model Decoding for LLM Inference via KV Cache Translation
PaperSource excerpt
“BridgeKV reduces response time from 39.80 s to 9.13 s, a 4.36× speedup over Llama-3.1-Instruct-8B-only inference. Throughput increases from 25.81 tok/s to 113.5 tok/s, a 4.40× gain, because decode follows the Llama-3.2-Instruct-1B anchor. ... Energy per request drops from 11.98 Wh (i.e., watt-hour) for Llama-3.1-Instruct-8B-only inference to 2.78 Wh for BridgeKV, corresponding to a 4.31× reduction.” · Show in paper
Verified
Tables 5 and 6 directly report these exact simulation numbers.
Efficient Memory Management for Large Language Model Serving with PagedAttention
PaperCodeSource excerpt
“Thus, recomputation is more efficient when the block size is small, while swapping is more efficient when the block size is large, though recomputation overhead is never higher than 20% of swapping’s latency. For medium block sizes from 16 to 64, the two methods exhibit comparable end-to-end performance.” · Show in paper
Verified
Figure 19 (a) and (b) directly support the microbenchmark and end-to-end performance comparison across block sizes.
FP4Quant: Efficient 4-Bit Communication Compression for Model-Parallel LLM Training
PaperSource excerpt
“Table 1 reports evaluation results on CMMLU and C-Eval (valid) with G = 64. FP4Quant consistently preserves downstream performance across model-parallel settings, achieving scores of 0.5287/0.4829 (TP), 0.5288/0.5342 (PP), and 0.5301/0.5253 (MP), compared with the BF16 baseline (0.5365/0.5215). Notably, under PP and MP configurations, FP4Quant matches or slightly exceeds BF16 on C-Eval, demonstrating strong capability preservation under ultra-low-bit compression.” · Show in paper
Verified
Table 1 confirms the reported scores and shows that Tahquant drops to near random-chance levels under TP and MP while FP4Quant remains close to BF16.
Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction
PaperCodeSource excerpt
“Across diverse long-context language and vision-language reasoning, and multi-turn dialogue benchmarks, our method substantially reduces KV memory while matching or surpassing full-cache inference. These results suggest that learned, globally calibrated KV eviction is not only a compression technique, but also a mechanism for improving long-context reasoning.” · Show in paper
Verified
The experimental results across Tables 1, 2, 3, 7 and Figure 5 directly support that DBTrimKV reduces KV memory while matching or surpassing full-cache inference across all tested benchmark categories.
fadebench· Method
PaperCodeSource excerpt
“All four are 4-bit GGUF builds served by llama-cpp-python with every layer on the GPU, and each file is pinned by its sha256” · Show in paper
A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation
PaperCodeSource excerpt
“Table 1 reports perplexity at n=1024 (T=512) on all five models. Every free-zone cell stays below the 0.2% threshold, with the largest drift at +0.17% (Qwen at 3×). The free zone holds at context lengths 512 and 2048 as well (Appendix B).” · Show in paper
Verified
All reported 2x and 3x perplexity values in Table 1 and Tables 4-6 are strictly below +0.2% relative drift.
How Inference Compute Shapes Frontier LLM Evaluation
PaperSource excerpt
“3.2. Generational gains are driven mainly by reach and reliability rather than efficiency” · Show in paper
Verified
The regression results in Table 4 and curve summaries in Table 9 support that reach and reliability improve more consistently across generations than token efficiency.
PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization
PaperSource excerpt
“Table 2 shows that PagedWeight preserves long-context quality with much lower memory consumption than FP16. At the 10 GB budget, PagedWeight reaches the same 17.0% average score as FP16 while using only 9.86 GB. In contrast, APL reaches only 12.2% at a similar memory consumption.” · Show in paper
Verified
Table 2 directly supports the claim, showing PagedWeight-10GB matches FP16's 17.0% average score and substantially outperforms APL-3bit on Passage Retrieval (17.5% vs. 10.5%) and NarrativeQA (10.0% vs. 4.9%).
How Inference Compute Shapes Frontier LLM Evaluation
PaperSource excerpt
“Newer models generally benefit less from parallel sampling. On all five benchmarks, the gain from parallel sampling decreases with model release date (Table 7; Figure 4B): TerminalBench (τ = −0.73), FrontierMath (−0.60), HealthBench (−0.47), HLE (−0.47), and most weakly SWE-Bench Pro (τ = −0.33).” · Show in paper
Verified
The tau values in the text match Table 7, though as the authors note elsewhere, each correlation is computed over only six models.