Search publications
SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference· Method
PaperCodeSource excerpt
“The routing summary projects token keys into a -dimensional space with a learned per-head matrix and averages them with surprisal weights : .” · Show in paper
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“Append m − 1 copies of η, and average the m cyclic permutations of the m pair registers.” · Show in paper
OpenAI Math GitHub Repository
PaperCodeSource excerpt
“Then EJ,T1,...,TJ−1,T |E(p,q)∼ω⟨fT (cT (p)) − mT,H(p) , gT (q)⟩| ≤ 4BB′ (D/K) 1/4 . Every realized history has at most MK−1 values. In particular, a history can be fixed before drawing the fresh test so that its average absolute discrepancy in (3.7) is at most the right side.” · Show in paper
Verified
The information-theoretic chain rule, Pinsker's inequality, and conditional independence steps are all sound.
TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate
PaperSource excerpt
“For any randomized quantization algorithm Q : → {0, 1}^{b·d} with bit-width b and any reconstruction map : {0, 1}^{b·d} → R^d, there exist a hard input instance x ∈ such that: Dmse(Q) := E[∥x − (Q(x))∥^2_2] ≥ 1/4^b.” · Show in paper
Verified
The lower bound proof in Theorem 3 (pages 14–15) is rigorous and follows directly from Yao's minimax principle, Lemma 3, and the pigeonhole principle.
GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
PaperCodeSource excerpt
“By contrast, we will aim to quantize the weights of all rows in the same order, and will show that this typically yields results with a final squared error that is similar to the original solutions. ... This reduces the overall runtime from O(drow · d^3_col) to O(max {drow · d^2_col, d^3_col}), i.e., by a factor of min {drow, dcol}.” · Show in paper
Verified
The complexity reduction follows from updating once per column, and empirical comparisons in Tables 1 and 8 support comparable accuracy to OBQ.
SoScale: Second-Order Scale Search for FP4 Quantization· Method
PaperSource excerpt
“Each activation block is quantized as xbt,b = QactF (xt,b; cactt,b), cactt,b = DF(xt,b), (14) where DF is the format-specific dynamic scale rule for activations. Because activation scales are computed online for every token, DF is kept lightweight and selects cactt,b for each token block using an element-wise MSE criterion.” · Show in paper
OpenAI Math GitHub Repository
PaperCodeSource excerpt
“Proposition 6.8 (Localization into current groups). For an active pattern in (11), there are poly(k) fixed search leaves such that, up to total absolute length-weighted error at most Ce^{-k}|R|, each leaf contributes \sum_{G \text{ current}} H_I(, , , ), = \overline{L}_{l,G,s}\mathcal{B}_{l,s}z_l. (14)” · Show in paper
Verified
The binary localization argument in Proposition 6.8 is clearly explained and all error terms are controlled by (13) and Lemma 6.1.
Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention
PaperSource excerpt
“LoRA-SVD protects a compact projection subspace in BF16 while retaining full-shape NVFP4 computation, reducing the linear-only loss gap from 1.40% to 0.61%.” · Show in paper
Verified
Table 3 and Section 4.2 directly support the reduction of the linear-only loss gap from 1.40% to 0.61% on 1B/20B-token pretraining.
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“We first bound the normalization of the successful event, then expose a common conditional density. We control entropy loss in two clipping steps, then deduce concentration near the endpoints of a fixed interval. Finally, one prefix and one threshold will satisfy every calibration and order requirement simultaneously.” · Show in paper
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“Use the axis-neighbor Metropolis chain on \mathcal{V}. Each of the 2e signed coordinate directions is proposed with probability q_B = 2^{-q_{\mathrm{bin}}}, q_{\mathrm{bin}} = \lceil \log_2(8d) \rceil; the unused probability, or a proposal leaving the cube, holds.” · Show in paper
AlignedServe: Orchestrating Prefix-aware Batching to Build a High-throughput and Computing-efficient LLM Serving System
PaperSource excerpt
“From the figure we conclude that, under the prefix-aware batching policy, more than 90% of iterations are able to complete the forward computing within 30ms, whereas the proportion of iterations that complete the forward computing within 30ms is less than 10% for the FCFS policy.” · Show in paper
Verified
The controlled ablation in Figure 13 and the preliminary experiment in Figure 3 directly support the claimed latency reduction.
Efficient Memory Management for Large Language Model Serving with PagedAttention· Method
PaperCodeSource excerpt
“In vLLM, we adopt the first-come-first-serve (FCFS) scheduling policy for all requests, ensuring fairness and preventing starvation. ... we implement an all-or-nothing eviction policy, i.e., either evict all or none of the blocks of a sequence. Furthermore, multiple sequences within one request (e.g., beam candidates in one beam search request) are gang-scheduled as a sequence group.” · Show in paper
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“Use the torus coordinates (x, z) = (u, v/u), (u, v) = (x, xz).” · Show in paper
OpenAI Math GitHub Repository
PaperCodeSource excerpt
“There is an extraction of a consistent tuple F′ of deterministic length ℓ′ = Θ(k), still in one of the two stream orientations, and a new discrete context C′ for which H(C′) ≤ Λ′ ≤ CqP log(w + 2)(σ + wP) + Cwσ, |domain of each slot of F′| ≤ Cq^d , Δ′ ≤ CK∗, 1 ≤ w ≤ Cσ^{1+η/2}/D.” · Show in paper
Verified
The standalone nature of the new context C' and the uniformity of all constants across admissible histories are carefully verified in Section 6.5 and the proof of Lemma 6.1 on page 47.
A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation
PaperCodeSource excerpt
“On GSM8K [41] with 8-shot chain-of-thought prompting [42] and exact-match scoring (n=1319 per cell), JoLT is near-lossless at 2× and 3× on both Mistral and LLaMA-2 (Table 7 in Appendix C). All free-zone deltas fall within their 95% confidence intervals of zero, with the largest shift at −1.21 pp (Mistral 3×).” · Show in paper
Verified
Tables 7 and 10 support the claims with the stated caveats (single model and single seed for LongBench).
FP4Quant: Efficient 4-Bit Communication Compression for Model-Parallel LLM Training
PaperSource excerpt
“The results demonstrate that FP4Quant consistently outperforms Tahquant in preserving model quality under FP4 communication compression. Under TP, Tahquant suffers from severe accuracy degradation (loss 3.4848, PPL = 32.6163), whereas FP4Quant with G = 64 achieves substantially lower loss (2.2306, PPL = 9.3050) and maintains a stable convergence trajectory close to the full-precision baseline, as illustrated in Fig. 5. Under PP, FP4Quant further demonstrates near-lossless performance (loss 2.1525, PPL = 8.6105), matching BF16 accuracy and outperforming Tahquant.” · Show in paper
Verified
The numerical metrics in Table 2 and convergence curves in Figure 5 support the claim for G = 64 (and G <= 256, as qualified on page 9).
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“Compressing the unknown block between these kernels gives a 2-by-2 matrix. The Hadamard equations bound its distance from sqrt(6) times its unitary polar factor. A closed cover of U(2), the group of 2-by-2 unitary matrices, therefore accounts for every feasible completion.” · Show in paper
TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate· Method
PaperSource excerpt
“Specifically, let Qmse be the quantization map corresponding to TurboQuantmse with a bit-width of b − 1. For any x ∈ the residual vector, defined as r := x − _mse (Qmse(x)), has a small L2 norm... We can then apply the QJL quantization map Qqjl on this residual vector, resulting in an overall bit-width of b” · Show in paper
ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMs
PaperCodeSource excerpt
“Figure 6 confirms substantial gains over FP16: Qwen2.5-7B achieves 2.0×–2.5× speedup (PRO 6000) and Llama 3.1-8B reaches 3.5× (RTX 5090), while memory usage drops by 1.5×–2.8×. Crucially, residual overhead is marginal: compared to uncompensated NVFP4, latency increases by only 3%–9%.” · Show in paper
Verified
The latency and memory numbers in Figure 6, Figure 8, and Table 10 support the reported speedup, memory reduction, and 3%–9% overhead over uncompensated NVFP4.
Towards Load-Aware Prefill Deflection for Disaggregated LLM Serving· Method
PaperCodeSource excerpt
“We perform a controlled microbenchmark in which we fix a decode batch of size B, with each decode request having K cached tokens, where K corresponds to the median prompt length of the workload (Table 1). We then inject chunked prefill tokens, increasing the total tokens processed per batch to chi, and measure the resulting step latency increase delta T_step(chi) across five successive chunks.” · Show in paper