Search publications
MomentKV: Closing the Directional Gap in KV Cache Eviction for Long-Context Inference
PaperSource excerpt
“The approximation error scales as O(\sigma^2) where \sigma = \max_{i\in\mathcal{E}} |\delta_i|, and a formal bound is given in Appendix A.” · Show in paper
Verified
The derivation in Appendix A.1-A.2 is mathematically sound and establishes both the O(\sigma^2) error bound and O(d^2) storage complexity.
AlignedServe: Orchestrating Prefix-aware Batching to Build a High-throughput and Computing-efficient LLM Serving System· Method
PaperSource excerpt
“The proposed serving system consists of three components, i.e., the KV pool, prefill instances and decoding instances, where both the prefill and decoding instances may be comprised of only one GPU, or multiple GPUs within the same server, or even many GPUs from different servers organized in a pipeline.” · Show in paper
earthdna
PaperCodeSource excerpt
“On SILVA 144 NR99 the 15 nearest neighbours in this space share the query's genus 87% of the time with exact search, the same as vsearch identity neighbours; longer k-mers add nothing once PCA is applied, and 128 components recover most of what 64 lose.” · Show in paper
Unverified
The manuscript states these benchmark results in the README text, but no evaluation data, tables, or script outputs are included to verify the percentages or comparisons.
A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation
PaperCodeSource excerpt
“Table 1 reports perplexity at n=1024 (T=512) on all five models. Every free-zone cell stays below the 0.2% threshold, with the largest drift at +0.17% (Qwen at 3×). The free zone holds at context lengths 512 and 2048 as well (Appendix B).” · Show in paper
Verified
All reported 2x and 3x perplexity values in Table 1 and Tables 4-6 are strictly below +0.2% relative drift.
SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference
PaperCodeSource excerpt
“The base LLM stays frozen, and the router adds under 0.1% parameters.” · Show in paper
Verified
Table 24 confirms parameter overhead is 0.054%–0.086% (<0.1%), and Table 3 confirms the 2.8–3.8 point drop when using training-free routing.
BridgeKV: Large-Model Prefill and Small-Model Decoding for LLM Inference via KV Cache Translation
PaperSource excerpt
“Table 3 compares the training cost of the KV-stitching module with that of the parameter-matched PEFT baseline, which fine-tunes the decode anchor alone with LoRA at the same number of trainable parameters as the stitched model. Training the module is cheaper in every setting.” · Show in paper
Verified
In all 16 rows of Table 3, the sum of Stage 1 and Stage 2 GPU-hours is strictly less than the PEFT baseline GPU-hours.
RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression· Method
PaperCodeSource excerpt
“In contrast, we propose hybrid sparse attention (HSA), which takes advantage of two-dimensional reduction, in both sequence and head dimensions together to achieve better estimation accuracy on KV token indices with top-k attention scores.” · Show in paper
RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
PaperCodeSource excerpt
“As shown in Figure 9, the best static split factor varies with different sequence lengths and token budgets, while adaptive compression decomposition provides comparable accuracy to the best static split factor in most cases.” · Show in paper
Verified
Figure 9 shows that adaptive RocketKV closely matches or exceeds the best static split factor across the tested benchmarks and token budgets.
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“At a node with allowance aw, independently for its children draw Ri uniformly from {1, . . . , B} and set ti = awRi/(BQ0), awi = λti.” · Show in paper
OpenAI Math GitHub Repository
PaperCodeSource excerpt
“Theorem 3.1 (Initial estimates). With the parameters and labeled interval tree of Section 2, there is a lazily evaluated random table U(0)(I, q) of integers in [0, N] such that, with probability at least 0.99, simultaneously at every state, c(I, q) ≤ U(0)(I, q) ≤ Ainitc(I, q). (3.1) Every zero-cost state has value zero on every execution. Conditional on any first request for an entry and the previously exposed randomness, its expected evaluation cost is at most (1 + |I|)2o(H), uniformly in the state. This cost bound includes arithmetic and random sampling and does not require (3.1) to hold.” · Show in paper
Verified
The proof handles unequal lengths, empty intervals, finite-precision dyadic arithmetic, and adaptive requests cleanly.
GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
PaperCodeSource excerpt
“We observe that, at 4 bits, GPTQ models reach only ≤ 0.25 lower perplexity than the full-precision versions, with a large gap to RTN results on OPT-175B. At 3-bit, RTN collapses, while GPTQ is still able to maintain good performance on most tasks, losing only 0.3 − 0.6 points for more than 5× compression.” · Show in paper
Verified
Table 5 confirms 4-bit perplexity differences (e.g., OPT-175B PTB: 12.26 vs. 12.01; BLOOM-176B PTB: 14.75 vs. 14.59) and 3-bit differences of 0.34-0.67 on WikiText2/C4/PTB (note BLOOM-176B PTB is 15.57 vs. 14.59, slightly above 0.6, which aligns with 'on most tasks').
BridgeKV: Large-Model Prefill and Small-Model Decoding for LLM Inference via KV Cache Translation
PaperSource excerpt
“In 12 of the 30 settings, the selected configuration uses only the first stage of training (Appendix C), so Md decodes with exactly the weights of its fine-tuned baseline, and BridgeKV still scores higher in all 12. In these settings, the gain comes from the stitched cache alone, not from extra trainable parameters on the decode side.” · Show in paper
Verified
Table 9 confirms 12 settings use S1 only, and Figure 3 confirms higher scores for BridgeKV in all of them.
TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate
PaperSource excerpt
“MSE defined as Dmse := Ex˜[∥x − x˜∥^2_2] is bounded by Dmse ≤ √3π/2 · 1/4^b for any b ≥ 0.” · Show in paper
Verified
The proof of Theorem 1 on page 11 is mathematically sound, using the rotational invariance of the L2 norm and the Panter-Dite high-resolution quantization formula.
QLoRA: Efficient Finetuning of Quantized LLMs
PaperCodeSource excerpt
“(2) Double Quantization, a method that quantizes the quantization constants, saving an average of about 0.37 bits per parameter (approximately 3 GB for a 65B model).” · Show in paper
Verified
The mathematical derivation on page 5 is exact, and Figure 3 confirms that Double Quantization preserves zero-shot accuracy.
AlignedServe: Orchestrating Prefix-aware Batching to Build a High-throughput and Computing-efficient LLM Serving System· Method
PaperSource excerpt
“To handle this situation, we set a time stamp for each internal node in the quad-tree, where the time stamp indicates how long the corresponding subtree has not generated a batch. If the time stamp associated with a subtree exceeds a given threshold, we give a higher priority to the subtree, thus avoiding potential starvation.” · Show in paper
M2XFP: A Metadata-Augmented Microscaling Data Format for Efficient Low-bit Quantization· Method
PaperCodeSource excerpt
“To implement this encoding, we first add a bias of 1 to the FP6 binary value, then clamp the result to ensure the high bits remain consistent with the original FP4 value. The lower 2 bits serve as metadata representing the additional mantissa precision.(Step 6 & 7)” · Show in paper
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“Given a transition matrix C and a nonnegative correction E, put D = C − E + EC. Then I − D = (I + E)(I − C). Thus a relation W = (I − C)p becomes (I + E)W = (I − D)p: the same vector p is represented by the new transition and the updated reward.” · Show in paper
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“Fix d ≥ 1 and put q = 4d^2. Choose a map h : Σ → [q] by assigning independent uniform labels to the letters, and apply it letter by letter to strings.” · Show in paper
SoScale: Second-Order Scale Search for FP4 Quantization
PaperSource excerpt
“Adding recovery to the MSE objective gives mixed gains over ScaleSearch: it is marginal under MXFP4, decreases NVFP4, and improves HiF4. In contrast, full SoScale is best for all three formats, indicating that recovery is most effective when coupled with Hessian-aware scale-code selection rather than a local MSE objective.” · Show in paper
Verified
The numerical results in Table 4 match the authors' characterization for all three formats.
Efficient Memory Management for Large Language Model Serving with PagedAttention
PaperCodeSource excerpt
“The improvement of vLLM over Orca (Oracle) on OPT-13B and the Alpaca dataset goes from 1.3× in basic sampling to 2.3× in beam search with a width of 6. Fig. 15 plots the amount of memory saving, computed by the number of blocks we saved by sharing divided by the number of total blocks without sharing. We show 6.1% - 9.8% memory saving on parallel sampling and 37.6% - 55.2% on beam search. In the same experiments with the ShareGPT dataset, we saw 16.2% - 30.5% memory saving on parallel sampling and 44.3% - 66.3% on beam search.” · Show in paper
Verified
Figures 14 and 15 directly support the Alpaca throughput and memory saving numbers. The ShareGPT memory saving numbers (16.2%-30.5% and 44.3%-66.3%) are reported in text without a corresponding figure, but are consistent with ShareGPT's longer prompts.