Search publications
Efficient Memory Management for Large Language Model Serving with PagedAttention
PaperCodeSource excerpt
“Our evaluations on various models and workloads show that vLLM improves the LLM serving throughput by 2-4× compared to the state-of-the-art systems [31, 60], without affecting the model accuracy at all. The improvements are more pronounced with longer sequences, larger models, and more complex decoding algorithms (§4.3).” · Show in paper
Verified
The experimental results across Sections 6.2-6.5 consistently demonstrate 1.67x-3.58x (and higher against Orca Max/FasterTransformer) throughput improvements at comparable normalized latencies, and PagedAttention is mathematically equivalent to standard attention.
Efficient Memory Management for Large Language Model Serving with PagedAttention
PaperCodeSource excerpt
“Indeed, our profiling results in Fig. 2 show that only 20.4% - 38.2% of the KV cache memory is used to store the actual token states in the existing systems.” · Show in paper
Verified
The profiling data in Figure 2 directly supports the 96.3% vs 20.4%-38.2% memory utilization comparison.
Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction· Method
PaperCodeSource excerpt
“Specifically, KV entries are stored in fixed-size pages, and each head maintains a block table for its currently active pages. Attention is then computed with variable-length kernels [Dao, 2023] using the resulting per-head sequence lengths.” · Show in paper
Efficient Memory Management for Large Language Model Serving with PagedAttention
PaperCodeSource excerpt
“Fig. 17 shows that vLLM can sustain 2× higher request rates compared to the three Orca baselines.” · Show in paper
Verified
Figure 17 directly supports the 2x higher sustainable request rate.
RULER: What's the Real Context Size of Your Long-Context Language Models?· Method
PaperCodeSource excerpt
“We evaluate all models using vLLM (Kwon et al., 2023), an LLM serving system with efficient KV cache memory management. For all models, we run the inference in BFloat16 on 8 NVIDIA A100 GPUs with greedy decoding.” · Show in paper
A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation· Method
PaperCodeSource excerpt
“For each layer, we compute the top eight token-mode singular vectors of its key cache and score consecutive layers by the mean squared cosine between their bases. A new group starts whenever the score falls below the median score across the network.” · Show in paper
Can LLMs reason over extended multilingual contexts? Towards long-context evaluation beyond retrieval and haystacks
PaperCodeDataSource excerpt
“Our extensive experiments with an open-weight large language model (LLM) reveal a pronounced gap between high- and low-resource languages, particularly for tasks requiring the model to aggregate multiple facts or predict the absence of information. We also find that, in multilingual settings, LLMs effectively utilize less than 30% of their claimed context length. Although off-the-shelf Retrieval Augmented Generation helps alleviate this to a certain extent, it does not solve the long-context problem.” · Show in paper
Verified
The experimental results in Tables 1, 5, 6 and Figures 5-8 consistently support the claimed gaps across languages, task types, and effective context length, as well as the partial mitigation by RAG.
Resource-Efficient Speculative Decoding for Long-Context LLM Serving· Method
PaperSource excerpt
“SpecStream uses NVIDIA Multi-Process Service (MPS) [19] to enable concurrent execution of the Target and Draft models in separate processes. ... The system applies a TPC mask in the separate Draft process so that its compute kernels run only on the designated set of TPCs [16], while preserving the Target model’s original access to compute resources.” · Show in paper
SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference· Method
PaperCodeSource excerpt
“At each decoding step, each head scores every span with an independent gate rather than a softmax, since expansion is a per-span decision: ” · Show in paper
GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
PaperCodeSource excerpt
“We observe that, at 4 bits, GPTQ models reach only ≤ 0.25 lower perplexity than the full-precision versions, with a large gap to RTN results on OPT-175B. At 3-bit, RTN collapses, while GPTQ is still able to maintain good performance on most tasks, losing only 0.3 − 0.6 points for more than 5× compression.” · Show in paper
Verified
Table 5 confirms 4-bit perplexity differences (e.g., OPT-175B PTB: 12.26 vs. 12.01; BLOOM-176B PTB: 14.75 vs. 14.59) and 3-bit differences of 0.34-0.67 on WikiText2/C4/PTB (note BLOOM-176B PTB is 15.57 vs. 14.59, slightly above 0.6, which aligns with 'on most tasks').
ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMs
PaperCodeSource excerpt
“On Llama 3.1-70B, ARCQuant consistently improves over the NVFP4+RTN baseline across all reported metrics, with gains of 0.65 in average zero-shot accuracy, 0.23 lower WikiText2 perplexity, and 0.50 higher MMLU, while maintaining a competitive margin to the FP16 baseline. We also validate ARCQuant on the MoE model Mixtral 8x7B-Instruct, where it again achieves consistent improvements over NVFP4+RTN, remaining close to FP16 performance.” · Show in paper
Verified
Table 9 confirms the exact reported gains on Llama 3.1-70B and Mixtral 8x7B-Instruct.
End-to-End Context Compression at Scale· Method
PaperCodeSource excerpt
“We segment the input into fixed-size chunks of 1,024 encoder tokens, compress each chunk, and assign it a segment identifier. The model receives the entire compressed sequence in a single prompt, together with one expand tool. At each turn, the model can issue a native tool call to expand with a segment identifier (Figure 6), and the tool returns the original text of that segment.” · Show in paper
GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers· Method
PaperCodeSource excerpt
“We address this problem by developing a quantized-matrix full-precision-vector product kernel which performs a matrix vector product by dynamically dequantizing weights when needed. Most notably, this does not require any activation quantization.” · Show in paper
SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference· Method
PaperCodeSource excerpt
“we compute during prefill and mark positions with as boundaries, where and are the sequence mean and standard deviation, and boundaries closer than tokens are merged, keeping the highest-surprisal one.” · Show in paper
FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving
PaperCodeSource excerpt
“FlashInfer’s fused kernel can yield 28 − 30% latency reduction under different settings (by changing the recent window size of Streaming-LLM). Original implementation is included as a baseline reference. We also show the kernel-level performance comparison between FlashInfer’s fused RoPE kernel and the combination of FlashAttention’s RoPE kernel and FlashAttention’s attention kernel. FlashInfer’s fused RoPE kernel achieves 1.6-3.7x higher bandwidth utilization compared to not fusing attention with RoPE” · Show in paper
Verified
Figure 9 data supports both the 28–30% end-to-end ITL reduction and the 1.6–3.7x kernel bandwidth utilization improvement.
Can LLMs reason over extended multilingual contexts? Towards long-context evaluation beyond retrieval and haystacks
PaperCodeDataSource excerpt
“Our results show that the effective context size is approximately 16k tokens for the best prompt-only and 32k tokens for the best RAG method. This suggests that the model is able to effectively use only up to 25% to 30% of its advertised context length.” · Show in paper
Verified
Table 1 directly supports the 16K and 32K ECW values under the authors' 30% degradation threshold.
How Inference Compute Shapes Frontier LLM Evaluation· Method
PaperSource excerpt
“We apply three deliberately simple inference-scaling techniques uniformly across every (benchmark, model, condition) cell: 1. Expanded total token budgets of 5M–30M tokens per trajectory (Table 2), sized to task complexity and one to three orders of magnitude above typical benchmark defaults. 2. Context compaction, which replaces earlier turns with a model-generated summary to enable serial scaling beyond the nominal context-window size. We adopt Inspect AI’s default summary-based strategy (UK AI Security Institute, 2024a), triggered when the running context exceeds 130k tokens (about 65% of the smallest 200k context window across models). 3. Iterative resubmission with a hard cap of 999 submissions per trajectory, allowing the model either to refine its previous answer or to try a substantially different approach between submissions. To reduce unproductive loops of near-identical submissions, we also employ a lightweight repetition guard that terminates a trajectory when a separate LLM judge finds three or more consecutive submissions semantically equivalent (Section A.1.3).” · Show in paper
A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation
PaperCodeSource excerpt
“On GSM8K [41] with 8-shot chain-of-thought prompting [42] and exact-match scoring (n=1319 per cell), JoLT is near-lossless at 2× and 3× on both Mistral and LLaMA-2 (Table 7 in Appendix C). All free-zone deltas fall within their 95% confidence intervals of zero, with the largest shift at −1.21 pp (Mistral 3×).” · Show in paper
Verified
Tables 7 and 10 support the claims with the stated caveats (single model and single seed for LongBench).
A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation· Method
PaperCodeSource excerpt
“The FlashJoLT decode kernel evaluates attention directly over the stored factors and packed residual, so no dense KV buffer is ever written to global memory.” · Show in paper
Towards Load-Aware Prefill Deflection for Disaggregated LLM Serving· Method
PaperCodeSource excerpt
“KAIROS is implemented on top of vLLM v0.18.1 [8] which supports disaggregated prefill/decode through a KV transfer API. KAIROS adds approximately 2000 lines of new code across the profiling for step latency models, the Deflection Dispatcher (which contains the analytical models and the deflection algorithm), the modifications to the scheduler on the decode nodes and plugins to report node state to the Deflection Dispatcher.” · Show in paper