Search publications
QQQ: Quality Quattuor-Bit Quantization for Large Language Models· Method
PaperCodeSource excerpt
“We adopt a layer-wise quantization framework GPTQ (Frantar et al., 2022), which advocates for a precise and efficient Hessian-based quantization compensation technique.” · Show in paper
FP4Quant: Efficient 4-Bit Communication Compression for Model-Parallel LLM Training
PaperSource excerpt
“Table 1 reports evaluation results on CMMLU and C-Eval (valid) with G = 64. FP4Quant consistently preserves downstream performance across model-parallel settings, achieving scores of 0.5287/0.4829 (TP), 0.5288/0.5342 (PP), and 0.5301/0.5253 (MP), compared with the BF16 baseline (0.5365/0.5215). Notably, under PP and MP configurations, FP4Quant matches or slightly exceeds BF16 on C-Eval, demonstrating strong capability preservation under ultra-low-bit compression.” · Show in paper
Verified
Table 1 confirms the reported scores and shows that Tahquant drops to near random-chance levels under TP and MP while FP4Quant remains close to BF16.
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“Trim a distance δp from both ends of p ′ , and convolve its indicator on the unit circle with the positive normalized kernel proportional to (sin(πNx)/(N sin(πx)))^{2s}” · Show in paper
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“Define on T3 U(t, ξ, z) = (η′(z)v1(t, ξ), η′(z)v2(t, ξ), −η(z) div_ξ v(t, ξ)).” · Show in paper
RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
PaperCodeSource excerpt
“We also propose a variant of RocketKV for multi-turn scenarios, which consistently outperforms other existing methods and achieves accuracy nearly on par with an oracle top-k attention scheme.” · Show in paper
Verified
Figure 6 and Table 4 on SCBench directly demonstrate that RocketKV-MT outperforms all non-oracle baselines and closely matches Exact-TopK across all tested token budgets.
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“We implement path sums, path increments, and threshold detection with link-cut trees [31].” · Show in paper
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“Templates and the transfer identity” · Show in paper
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“Let δj (i), 0 ≤ j < 10, be item i’s coefficient in these successive left sides, and define its integer primary score by p(i) = ∑_{j=0}^9 100^j δj (i).” · Show in paper
KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
PaperCodeSource excerpt
“From Table 5, we observe that group sizes 32 and 64 yield similar results, whereas the performance significantly decreases when the group size reaches 128.” · Show in paper
Verified
Tables 5, 6, and 7 directly support the stated sensitivity to group size 128 and the robustness across residual lengths 32 and 128.
RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression· Method
PaperCodeSource excerpt
“The core idea in RocketKV-MT is to avoid permanently evicting any KV tokens during the first stage; instead, all KV tokens are retained in memory across turns, ensuring that no potentially useful context is lost. Meanwhile, to preserve computational efficiency, the second stage still performs dynamic selection over the subset of KV tokens filtered by the first stage, similar to the original RocketKV.” · Show in paper
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“In every port coordinate independently, let κ_b0 be the law of a sum of six uniform variables on [−b0/6, b0/6]. Convolve IG in its port variables with the product of these kernels, writing the result as I_tilde_G.” · Show in paper
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“At a nonleaf I, consider all cones q |-> Ue(I, r) + L0|q − r|1 whose centers are grid states at some scale b with b/2 ≤ U(I, r) ≤ 4Ab.” · Show in paper
OpenAI Math GitHub Repository
PaperCodeSource excerpt
“Proposition 4.7 (Reward identity and a shallow target). For every stage, ClPcl = Pcl − Bl(I − S). The vectors Wl = Ble satisfy 0 ≤ Wl ≤ 1, Wl = Pclp0 − ClPclp0, and their updates are Wl+1(xi) = ri(x)(Wl(x) + (ElWl)(x)). If ¯xl is the all-primary copy of a base vertex x, then 0 ≤ p0(x) − Wl(¯xl) ≤ |V0| 2−(H−1)l.” · Show in paper
Verified
The algebraic identity and the active-column counting argument are rigorous.
RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
PaperCodeSource excerpt
“Again, RocketKV shows robust performance and a clear advantage over all other methods across various token budget and sequence length settings. Overall, we can see that the accuracy loss of RocketKV is negligible under short sequence lengths and gradually becomes larger as sequence length increases.” · Show in paper
Verified
Figure 5(g-r) and the detailed RULER tables (Tables 6-9, 12-15, 18-21) confirm RocketKV's advantage across sequence lengths and token budgets.
PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization
PaperSource excerpt
“Across several memory-sensitive MoE serving scenarios, PagedWeight improves the quality-memory tradeoff over several existing quantization baselines. PagedWeight achieves FP16-equivalent accuracy with up to 72.0% GPU memory savings and 1.94× throughput improvement, and improves quality over quantization methods by up to 39.3% at a similar memory budget with at most 4.1% throughput loss.” · Show in paper
Verified
The reported figures match the experimental tables: (35.25 - 9.86)/35.25 = 72.0% memory savings, 130.1/67.1 = 1.94× throughput, (17.0 - 12.2)/12.2 = 39.3% quality gain, and (401.9 - 385.4)/401.9 = 4.1% max throughput loss.
Bursts That Outlast Themselves: Persistent Degradation in Prefix-Cached Agent Serving
PaperSource excerpt
“We show that this makes prefix-cached agent serving metastable. With an identical base workload, a few minutes of extra arrivals leaves a vLLM server in a degraded regime—a ∼10% hit ratio, about half the throughput, and a backlog that grows for the remaining 70 minutes—while its no-burst twin stays healthy (9 of 12 pairs; none in the reverse direction).” · Show in paper
Verified
The paired experiments, no-cache and session-cap controls, and reproductions across models and engines directly support the claim of metastability.
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“For a finite list L in X, let H_L^pi be its count vector on the parts of pi, with repetitions counted. Then H_alpha(L) = bigoplus_pi Pr[pi] H_L^pi (77) is a deterministic finite-dimensional map whose distance equals E||H_L^pi - ^pi||_1.” · Show in paper
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“For the rest of the proof put δ = min(q, 1/2), Λ = , = δ^3 log n. We use Tr(w) = D_δ(w) throughout the construction. The requested channel q is obtained by deleting each surviving bit of a δ-trace with additional probability (q − δ)/(1 − δ). Thus all upper bounds on distinguishability proved at δ also hold at q.” · Show in paper
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“A triangular cell is specified by polynomials Ch(u, b<h) ∈ , deg_wt Ch ≤ h, bounded centers lh, and widths 0 < wh ≤ 1/32.” · Show in paper
GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers· Method
PaperCodeSource excerpt
“Concretely, we apply the algorithm to B = 128 columns at a time, keeping updates contained to those columns and the corresponding B × B block of H−1 ... For smaller models, applying dampening, that is adding a small constant λ (we always choose 1% of the average diagonal value) to the diagonal elements of H ... Algorithm 1 Quantize W given inverse Hessian H−1 = (2XX> + λI)−1 and blocksize B.” · Show in paper