Search publications
Gemma 4 Technical Report
PaperSource excerpt
“On the 150M image encoder, quantizing activations and weights to 8-bit precision (W8A8) yields a 2x reduction in total forward-pass memory footprint (from 400 MB to 200 MB, including on-device compilation overhead) and a 44% reduction in on-device latency relative to Gemma 3n on newer hardware. On the audio encoder, we further reduce activation precision to 8 bits and weight precision to {2, 4, 8} bits, varying by layer cluster. Overall, we achieve a 78% reduction in on-disk footprint, from 390 MB in Gemma 3n to 87 MB in this version.” · Show in paper
Unverified
While the 78% audio encoder on-disk footprint reduction (390 MB to 87 MB) is corroborated by Table 7, no benchmark table or hardware measurement details are provided to verify the 2x forward-pass memory reduction or 44% on-device latency reduction for the 150M image encoder.
Resource-Efficient Speculative Decoding for Long-Context LLM Serving
PaperSource excerpt
“SpecStream achieves average throughput speedups of 1.40× and 1.23× over TriForce on Qwen3 and InternLM2.5, respectively.” · Show in paper
Verified
Figure 6 and Section 5.2.2 support both the throughput speedups over TriForce and the nuanced limitation regarding Draft KV Cache residency.
RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
PaperCodeSource excerpt
“Based on the figure, we can see that RocketKV consistently outperforms all other methods, especially under lower token budgets. For Llama3.1-8B-Ins, RocketKV achieves almost no accuracy loss with a token budget of 512 and above, and only 1.1% average accuracy drop with a token budget of 256.” · Show in paper
Verified
Table 2, Table 10, Table 16, and Figure 5(a-c) confirm RocketKV's consistent advantage at low token budgets and the 1.1% (52.2 to 51.1) drop at budget 256 on Llama3.1-8B-Ins.
OpenAI Math GitHub Repository
PaperCodeSource excerpt
“J̄(V) − J̄(U) ≤ Cm + (m/Nk)(C + B/h0) + B/q. When B ≤ d^2, the right side is at most C'd for an absolute constant C'.” · Show in paper
Verified
All integrability conditions (Lemma 6.2) and independence/conditioning steps are carefully verified.
MomentKV: Closing the Directional Gap in KV Cache Eviction for Long-Context Inference
PaperSource excerpt
“To verify that half-precision accumulation does not degrade quality in practice, we compare the LongBench scores obtained with FP16 and FP32 moment statistics on LLaMA-3.1-8B at L=128. The average score difference is less than 0.05 across all task categories, confirming that the numerical error introduced by half-precision accumulation is negligible relative to the approximation error O(\sigma^2).” · Show in paper
Verified
The reported experiment in Appendix C.2 supports the claim, though no per-category table is shown.
OpenAI Math GitHub Repository
PaperCodeSource excerpt
“Theorem A.1 (Fixed-length binary conversion). There is an absolute integer M0 such that, for every integer M >= M0 and every alphabet Gamma with 1 <= K = |Gamma| <= M, there exists a map c : Gamma -> {0, 1}^P with P = M^2 whose concatenation extension enc satisfies (P/804) ED(x, y) <= ED(enc(x), enc(y)) <= P ED(x, y) (x, y in Gamma^M).” · Show in paper
Verified
The probabilistic existence of the codebook (Lemma A.2), the span-trimming bound for unequal labels, and the bipartite multigraph decomposition for equal labels are all rigorous.
Towards Load-Aware Prefill Deflection for Disaggregated LLM Serving
PaperCodeSource excerpt
“Both terms come from the predictors in §3.3, which on held-out traces achieve MAPE < 10%. Predictions are O(1) via simple linear regressors.” · Show in paper
Verified
The paper consistently reports MAPE < 10% on held-out traces for the linear regression step-latency models.
ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMs
PaperCodeSource excerpt
“Our analysis confirms that, for compensated outlier channels, the dual-stage mechanism matches the worst-case error bounds of standard MXFP8.” · Show in paper
Verified
The mathematical derivation in Section 3.4 is sound and correctly shows that is bounded by , which is tighter than MXFP8's bound.
Gemma 4 Technical Report· Method
PaperSource excerpt
“Our local to global attention ratio patterns follow Gemma Team (2025a), that is, 4-to-1 local attention blocks for E2B and 5-to-1 for the rest. We improve memory efficiency by re-using keys as values in the global attention layers (except in E2B and E4B), i.e. , values = keys. We encode position with p-RoPE with p = 0.25 on global attention layers and with RoPE on local attention layers, effectively reducing the global KV cache by 37.5%. The RoPE frequencies are set to 1M and 10k on global and local attention layers, respectively. Finally, we share the KV cache with ratios of 20/35 and 18/42 for the E2B and E4B model.” · Show in paper
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“For integers p and q > 0, replace each weight by wp/q(e) = qw(e) − p. ... Choose K = 1 + ⌈log2(2(Win + 1)n^2)⌉, computable from integer bit lengths. For each vertex, start with the closed interval [−Win, Win] and make K bisections.” · Show in paper
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“At a real root \alpha of a nonzero P, the associated weighted sum is S(\alpha) = \sum_{j=1}^d w_j \operatorname{sgn} b_j(\alpha).” · Show in paper
AgentPerfBench: A Benchmarking and Evaluation Suite for Inference Performance of Agentic LLMs
PaperSource excerpt
“The longer agentic profiles exceed the single-H100 HBM budget: coding agent exceeds the wall by C = 40 and reaches 155 GB at C = 80, while terminal reaches 126 GB (Appendix Figure 4b clips both at the 80 GB wall).” · Show in paper
Verified
The analytical CF model in Figure 4b and Table 14b directly supports the capacity saturation thresholds.
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“Replace each schema by a path with 4(r + 1) private internal types, each of dimension one. Call this subdivided path a wire; its endpoints are the original real types, and its segments are links.” · Show in paper
M2XFP: A Metadata-Augmented Microscaling Data Format for Efficient Low-bit Quantization
PaperCodeSource excerpt
“This Pareto analysis reveals a crucial asymmetry: element-level metadata is superior under a fixed shared scale due to its ability to capture dominant outliers without global adjustments, while subgroup-level metadata becomes preferable once an adaptive shared scale is incorporated.” · Show in paper
Verified
Figures 6 and 7 clearly demonstrate the crossover in MSE between Elem-EM and Sg-EM-2bit at 4.5–4.75 EBW when switching from fixed to adaptive shared scale.
End-to-End Context Compression at Scale
PaperCodeSource excerpt
“At 64k tokens, the 16x LCLM reaches the first token faster than the smallest uncompressed models (0.21 s, compared with 0.81 s for Llama 3.2 1B and 0.97 s for Qwen3-0.6B) while being far more accurate on LongHealth5 (66.25% vs. 22.50% and 32.75%). The 4x LCLM exceeds the uncompressed Qwen3-4B-Instruct-2507 on LongBench with a 6x lower TTFT.” · Show in paper
Verified
The vLLM-served benchmarks in Figure 5 and Section 6.2 directly substantiate the reported TTFT and accuracy comparisons.
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“Approximate this pair in L^1 by rational-valued step functions on finite dyadic rectangular grids. For example, conditional expectations on the successive dyadic grids converge in L^1, and subsequent rational quantization preserves the range [0, 1]^2 and the convergence.” · Show in paper
Efficient Memory Management for Large Language Model Serving with PagedAttention
PaperCodeSource excerpt
“Thus, recomputation is more efficient when the block size is small, while swapping is more efficient when the block size is large, though recomputation overhead is never higher than 20% of swapping’s latency. For medium block sizes from 16 to 64, the two methods exhibit comparable end-to-end performance.” · Show in paper
Verified
Figure 19 (a) and (b) directly support the microbenchmark and end-to-end performance comparison across block sizes.
SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
PaperCodeSource excerpt
“In Table 1, we verify the assumption that simulated per-channel activation quantization successfully bridges the accuracy with the FP16 baseline, which echos the findings of Bondarenko et al.. However, per-channel activation quantization does not map well to hardware-accelerated GEMM kernels, that rely on a sequence of operations executed at a high throughput (e.g., Tensor Core MMAs) and do not tolerate the insertion of instructions with a lower throughput (e.g., conversions or CUDA Core FMAs) in that sequence.” · Show in paper
Verified
Table 1 supports the accuracy of simulated per-channel quantization, and the hardware GEMM constraint is a well-established architectural fact.
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“To carry this table through another pair of splits, encode a payload v ∈ Ui and a work-register value m ∈ M as a single basis vector. Set Ui+1 = F Ui×M 2 with basis [v, m].” · Show in paper
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“Lemma A.2 (Separation of unequal-label segments). For sufficiently large integers M and integers 1 <= K <= M, there are K binary words of length P = M^2 such that any two length-a segments of distinct codewords satisfy LCS <= 0.99a whenever a >= a0 := ceil(40 log2(MP)) = ceil(120 log2 M).” · Show in paper