Search publications
TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate
PaperSource excerpt
“Our data-oblivious algorithms, suitable for online applications, achieve near-optimal distortion rates (within a small constant factor) across all bit-widths and dimensions.” · Show in paper
Verified
The theoretical upper and lower bounds are rigorously proved in Theorems 1–3, and the empirical distortion plots in Figure 3 align with these bounds.
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“Work in the value Hilbert space \mathcal{H} = \mathbb{C} \oplus \ell^2(\mathcal{V}), and define the column on its birth cell by \Phi_v(y) = (\varphi_v(y), \sqrt{\epsilon_v}\mathbf{e}_v), y \in J_v.” · Show in paper
End-to-End Context Compression at Scale
PaperCodeSource excerpt
“As shown in Figure 6, our agent substantially improves performance over the same LCLM with 16x compressed context and no expansion.” · Show in paper
Verified
Table 36 and Figure 6 provide exact per-task and average accuracy numbers showing large gains (+18.50, +23.19, and +25.50 points at 4k, 8k, and 16k).
AlignedServe: Orchestrating Prefix-aware Batching to Build a High-throughput and Computing-efficient LLM Serving System
PaperSource excerpt
“As the CDF of time to schedule an iteration shown in Figure 11 presents, more than 95% of iterations in AlignedServe are scheduled within 5 ms, in contrast to DistServe, where 80% of iterations take more than 10 ms.” · Show in paper
Verified
Figure 11 directly measures iteration scheduling latency and supports the stated >95% within 5ms vs. DistServe's 80% >10ms.
MomentKV: Closing the Directional Gap in KV Cache Eviction for Long-Context Inference· Method
PaperSource excerpt
“Applying the first-order Taylor expansion exp(\delta_i) \approx 1 + \delta_i, the zero-sum property \sum_{i\in\mathcal{E}} \delta_i = 0 collapses the denominator to n_e. In the numerator, the constant term produces the mean evicted value \bar{v} = n_ \sum_{i\in\mathcal{E}} v_i, and the linear term produces a matrix-vector product involving the empirical value-key covariance \tilde{S} = \sum_{i\in\mathcal{E}} v_i (k_i - \bar{k})^\top \in \mathbb{R}^{d\times d}. Dividing by n_e yields the closed-form approximation \hat{f}_{\mathcal{E}}(q) = \bar{v} + \frac{1}{n_e \sqrt{d}} \tilde{S} q.” · Show in paper
AlignedServe: Orchestrating Prefix-aware Batching to Build a High-throughput and Computing-efficient LLM Serving System· Method
PaperSource excerpt
“We maintain a tuple (request counter, block counter) at each internal node to characterize the corresponding sub-tree, where the request counter denotes the total number of in-flight requests kept in the sub-tree, and the block counter denotes the total number of memory blocks consumed by the KVCache of these in-flight requests.” · Show in paper
A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation· Method
PaperCodeSource excerpt
“We apply a random orthogonal rotation along the feature axis to spread residual energy evenly across channels, so that no single outlier channel dominates the quantizer’s dynamic range. The rotation matrix is the dh × dh factor from the QR decomposition of a seeded Gaussian, shared by every cell and regenerated from its seed at decode, so it costs no storage. We then quantize the rotated entries uniformly at b ∈ {0, 2, 4, 8} bits per element with one fp32 scale per row, where b=0 means no residual is stored.” · Show in paper
RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression· Method
PaperCodeSource excerpt
“For an overall compression ratio of c, we define a split factor r so that c is split into c^r for the first stage and c^(1-r) for the second stage, where 0 <= r <= 1. We use the following formula to adaptively determine r: r = min(0.2 + 0.06 * log2(c), 0.8)” · Show in paper
FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving· Method
PaperCodeSource excerpt
“FlashInfer guarantees both attention and contraction stage are compatible with CUDAGraphs (Gray, 2019; Nguyen et al., 2021). Both attention and contraction stage use persistent kernel and the grid size is fixed once compiled, which means the kernel is launched with the same grid size for each generation step. We set the fixed offset for each section of the workspace buffer to store partial outputs and plan information to make sure the pointers passed to the kernel are the same for each generation step, meeting the requirement of CUDAGraphs” · Show in paper
Bursts That Outlast Themselves: Persistent Degradation in Prefix-Cached Agent Serving
PaperSource excerpt
“A budgeter that bounds the resident KV demand of active sessions to the pool, with thresholds that need no per-workload tuning, prevented the failure in every run we tested.” · Show in paper
Verified
Prevention runs across triggers and the session-mix shift experiment (Figure 6) support the claim.
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“For every ordered pair of distinct vertices p, q in B, consider the two thresholds d(p, q)/100, Jd(p, q). For each threshold tau, protect the entry immediately following a drop from a distance strictly greater than tau to a distance at most tau in the list d(p, A1), d(p, A2), . . . , d(p, Am)” · Show in paper
BridgeKV: Large-Model Prefill and Small-Model Decoding for LLM Inference via KV Cache Translation
PaperSource excerpt
“Across six model pairs and five datasets, BridgeKV outperforms fine-tuned small-model baselines with matched trainable parameter budgets in all 30 settings. It closes up to 80% of the quality gap to using the larger model for both phases. On one A40 GPU, Llama-3.1-Instruct-8B→Llama-3.2-Instruct-1B achieves 3.0× faster requests than the larger model alone, with latency close to the smaller model alone. BridgeKV can improve the hardware and system solutions that separate prefill and decode phases across GPUs. A separate serving simulation models request arrivals, queues, and cache transfer. For Llama-3.1-Instruct-70B→Llama-3.2-Instruct-1B, it predicts up to 33.86× faster requests and 27.5× lower energy use than using the larger model for both prefill and decode phases.” · Show in paper
Verified
All components of the overarching claim are directly supported by Figure 3, Figure 1, Section 4.3, and Table 7.
Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving
PaperCodeSource excerpt
“KIVI4 provides the most stable quality across models” · Show in paper
Verified
Table 2 confirms that KIVI4 avoids the large accuracy drops seen in TurboQuant, SnapKV, and CaM on summarization while maintaining competitive or superior scores on QA and few-shot tasks.
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“For reciprocal supports, arrange the representative pairs in time order and apply the geometric description along a balanced binary tree. Each successful node restricts one endpoint domain for each child. Fresh validation samples bound the loss from each restriction before any later success is imposed. An additive potential bounds the total message length.” · Show in paper
AlignedServe: Orchestrating Prefix-aware Batching to Build a High-throughput and Computing-efficient LLM Serving System
PaperSource excerpt
“Experimental results demonstrate that the fraction of iterations that contain requests from different batches is no more than 8.61% and 12.37% on ShareGPT and LongBench, respectively.” · Show in paper
Verified
The reported measurements in Section 4.4 and Figure 15 substantiate the bounded overheads.
M2XFP: A Metadata-Augmented Microscaling Data Format for Efficient Low-bit Quantization
PaperCodeSource excerpt
“Crucially, we find that simply preserving the maximum element of the block in FP16 precision drastically reduces MXFP4’s perplexity, nearly matching FP4 and NVFP4. This experiment confirms that the mishandling of block maximum is the primary weakness of MX quantization.” · Show in paper
Verified
Figure 3 directly supports the claim that preserving the group-wise maximum in FP16 drastically reduces MXFP4 perplexity on LLaMA3-8B and LLaMA3-70B.
BridgeKV: Large-Model Prefill and Small-Model Decoding for LLM Inference via KV Cache Translation
PaperSource excerpt
“The benefit of BridgeKV is more pronounced when there is a large gap between the prefill anchor and decode anchor, especially for output-heavy workloads. As it is shown in Table 7, the benefit can go from 1.75× (1.76×) to 33.86× (27.5×) in terms of response time and energy savings.” · Show in paper
Verified
Table 7 directly supports the reported range of response speedups and energy savings.
SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference
PaperCodeSource excerpt
“SEKV gains more from extra budget than ShadowKV, so its gap to ShadowKV narrows from 3.4 points at 5% to 1.0 at 20% on RULER and from 2.8 to 0.3 on NIAH, and on LLAMA-3.1-8B SEKV matches ShadowKV on NIAH at 20%.” · Show in paper
Verified
Table 4 directly supports all stated numbers.
FP4Quant: Efficient 4-Bit Communication Compression for Model-Parallel LLM Training· Method
PaperSource excerpt
“Adjacent threads exchange their codes through a stride-1 butterfly shuffle: hii = __shfl_xor_sync(0xffffffff, qk,i, 1) & 0x0F. (4) For even lanes (i mod 2 = 0), the shuffled value provides the high nibble from the adjacent thread, while the local code remains as the low nibble. Each even lane then assembles one byte as b = (hii ≪ 4)| qk,i and performs a single coalesced store, whereas odd lanes remain inactive.” · Show in paper
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“For a pivot z = , retain only its label (z), and replace it by z' = \rho_{i_0}((z)).” · Show in paper