Search publications
Efficient Memory Management for Large Language Model Serving with PagedAttention· Method
PaperCodeSource excerpt
“Specifically, PagedAttention partitions the KV cache of each sequence into KV blocks. Each block contains the key and value vectors for a fixed number of tokens, which we denote as KV block size (B).” · Show in paper
Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction· Method
PaperCodeSource excerpt
“Specifically, KV entries are stored in fixed-size pages, and each head maintains a block table for its currently active pages. Attention is then computed with variable-length kernels [Dao, 2023] using the resulting per-head sequence lengths.” · Show in paper
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“Choose the integer base Bπ = 2⌈(η−Hπ)n⌉ and use the digit ranges 0 ≤ h < ⌈P/Bπ⌉, 0 ≤ ℓ < Bπ.” · Show in paper
AlignedServe: Orchestrating Prefix-aware Batching to Build a High-throughput and Computing-efficient LLM Serving System
PaperSource excerpt
“We give a deep insight into the decode within each iteration via both theoretical analysis and trace-driven experiments, demonstrating that tokens generated in the same iteration may have different costs depending on the lengths of their prefixes. The disparity between tokens introduces iteration-level bubbles, which will degrade the overall performance significantly.” · Show in paper
Verified
Both the theoretical analysis and the empirical vLLM measurements on H100 GPUs directly substantiate that long-prefix tokens create iteration-level bottlenecks.
A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation· Method
PaperCodeSource excerpt
“For each layer, we compute the top eight token-mode singular vectors of its key cache and score consecutive layers by the mean squared cosine between their bases. A new group starts whenever the score falls below the median score across the network.” · Show in paper
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“We prove Theorem 3.2 by adapting the local-form construction of [22, Sections 3–10]. The scalar analytic and finite combinatorial estimates remain those of H. The row estimates of Section 4 replace the uses of one original input at different depths.” · Show in paper
Towards Load-Aware Prefill Deflection for Disaggregated LLM Serving· Method
PaperCodeSource excerpt
“At deployment time, a one-time profiling script collects data for the analytical models by running production traces at various request rates against the target model hardware combination. The step latency and batch contents are recorded at each step and used to train the analytical models. Total profiling time is ~30 minutes per model/hardware combination.” · Show in paper
Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention
PaperSource excerpt
“For the second moment, direct NVFP4 gives 0.0116, whereas the complete pipeline gives 0.0053, a 54.3% reduction. ... In separate 1B native-state ablations, retaining NVFP4 state iteration while removing any required preprocessing stage caused rapid instability or divergence, so no comparable endpoint loss existed. To our knowledge, this is the first demonstration of stable NVFP4 AdamW momentum-state storage in an integrated 1B pretraining run.” · Show in paper
Verified
The reconstruction error reductions, bit-width accounting, and 1B/20B-token pretraining stability are all directly supported by the reported data.
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“Build a balanced binary tree on it from Section 5: its root is the middle window, and recurse on the earlier and later halves. Each node has its original supports A, B from Lemma 5.5.” · Show in paper
SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference
PaperCodeSource excerpt
“a single global threshold costs about 2 points, supporting per-head thresholds.” · Show in paper
Verified
Table 3 directly supports the ~2-point cost (1.9 to 2.6 points across the four settings).
Bursts That Outlast Themselves: Persistent Degradation in Prefix-Cached Agent Serving
PaperSource excerpt
“In the degraded regime nearly every turn misses and the server runs at its no-cache capacity. A first-order workload-balance condition, with per-miss cost calibrated independently on cache-disabled runs, explains when the regime cannot drain (§4).” · Show in paper
Verified
The independently calibrated sessions/s falls squarely within the empirically observed break-even bracket of 0.010–0.014 sessions/s.
M2XFP: A Metadata-Augmented Microscaling Data Format for Efficient Low-bit Quantization· Method
PaperCodeSource excerpt
“The optimization first finds the optimal mantissa refinement k for each subgroup given a bias b, then selects the best group-level exponent bias b.” · Show in paper
AQLoRA: A Zero-Search Recipe for Fast Quantized LoRA Fine-Tuning
PaperCodeSource excerpt
“We also compare our rule against the only deployed alternative we know of: Unsloth’s released dynamic-4bit selection for Qwen2.5-7B, a hand-curated list covering 34 of the 196 linear layers. Substituting their list for ours in an otherwise identical run gives 89.0 on the 8-task average, matching our 89.0.” · Show in paper
Verified
The direct substitution experiment on Qwen2.5-7B supports the parity claim.
PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization
PaperSource excerpt
“Table 2 shows that PagedWeight preserves long-context quality with much lower memory consumption than FP16. At the 10 GB budget, PagedWeight reaches the same 17.0% average score as FP16 while using only 9.86 GB. In contrast, APL reaches only 12.2% at a similar memory consumption.” · Show in paper
Verified
Table 2 directly supports the claim, showing PagedWeight-10GB matches FP16's 17.0% average score and substantially outperforms APL-3bit on Passage Retrieval (17.5% vs. 10.5%) and NarrativeQA (10.0% vs. 4.9%).
A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation
PaperCodeSource excerpt
“At 2×, JoLT drifts +0.01% while the best low-rank baseline (KQ-SVD) drifts +5.27%. Fixed-rate 8-bit quantization (int8 per-channel and fp8 e4m3) is also near-lossless at 2×, but has no operating point above 2×. Near 4×, the dedicated quantizers at their native ratios outperform JoLT: TurboQuant at +0.28% and KIVI at +1.70% against JoLT’s +5.65%. JoLT’s advantage over every evaluated baseline therefore lies in the band above 2× and below roughly 4×, which is exactly the free zone (Figure 2b).” · Show in paper
Verified
Tables 2, 14, 15, and 16 directly support this comparison, and the authors transparently note reimplementation deviations in Appendix E.
A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation
PaperCodeSource excerpt
“Per-group rank allocation beats uniform rank at all 15 grid cells (ratios 2, 3, 4, 6, 8 at three context lengths), with a peak advantage of 0.97 PPL at n=512, 4×.” · Show in paper
Verified
Tables 12 and 13 confirm all reported perplexity differences.
TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate
PaperSource excerpt
“Inner-product distortion defined as Dprod := Ex˜[|⟨y, x⟩ − ⟨y, x˜⟩|^2] is bounded by Dprod ≤ √3π^2 · ∥y∥^2_2 / d · 1/4^b for any b ≥ 0.” · Show in paper
Verified
The proof on pages 13–14 correctly combines the conditional expectation/variance of QJL (Lemma 4) with the MSE bound of Theorem 1 at bit-width b-1.
SoScale: Second-Order Scale Search for FP4 Quantization
PaperSource excerpt
“We identify an outlier-induced scale mismatch in hierarchical FP4 quantization: default range-preserving scales can reserve scarce FP4 levels for regions that receive no assignments, reducing resolution for dense weight values.” · Show in paper
Verified
Figure 1a-c directly illustrates the unoccupied extreme levels under default scaling, the reduced element-wise error in the dense region, and the perplexity reduction achieved by SoScale compared to ScaleSearch and RTN.
FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving· Method
PaperCodeSource excerpt
“We developed CUDA/CUTLASS (Thakkar et al., 2023) templates for FlashAttention, designed specifically for both dense and block-sparse matrices and compatible with NVIDIA GPU architectures from Turing to Hopper (sm75 to sm90a).” · Show in paper
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“On the grid tℓ = ℓ/(20n), 0 ≤ ℓ < 20n, form the real matrix with R = 40n rows A2ℓ,j = ℜ(dje(jtℓ)), A2ℓ+1,j = ℑ(dje(jtℓ)). ... We round p to a vector in {0, 1}n, preserving an upper bound on its total mass.” · Show in paper