Search publications
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“For each integer b ≥ 1, put c = 4/(1 − q) , Nb = 2⌈log2(cq−b)⌉, Γb = N−1b Z. The periodic grid key of z ∈ R is Jb(z) = ⌊Nb{z}⌋, where {z} = z − ⌊z⌋.” · Show in paper
OpenAI Math GitHub Repository
PaperCodeSource excerpt
“Theorem 10.4 (Charged expected running time). The large-input algorithm has worst-case expected running time N1+o(1) on the logarithmic-word RAM, over its internal randomness. This includes input processing, preprocessing, initial and refined table calculations, all rational arithmetic, sampling, and data-structure operations. Its workspace on every execution is bounded by a fixed polynomial in N.” · Show in paper
Verified
The dependency counting and conditioning arguments hold unconditionally, even on executions with inaccurate seeds or failed group samples.
How Small Can You Go? A Controlled Study of LoRA Rank, Target Modules, and Quantization Trade-offs for Text-to-SQL on a 60M-Parameter Model
PaperCodeSource excerpt
“Expanding from {q,v} at r=8 (53.4%, 344K params) to the full attention set {q,k,v,o} (59.2%, 590K params) recovered 5.8 points at the cost of 71% more trainable parameters and a 6.3% memory increase (1.59 to 1.69 GB). Adding feed-forward layers {q,v,FFN} (58.6%, 786K params) matched {q,k,v,o} in accuracy but required 33% more parameters, an unfavourable trade-off. Notably, LoRA r=16 on {q,v} (59.6%, 590K params) matched {q,k,v,o} at the same parameter budget, suggesting that, in this setting, rank is a more efficient lever than module expansion.” · Show in paper
Verified
Table 2 directly supports the exact accuracy, parameter count, and memory comparisons between module expansion and rank increase.
SoScale: Second-Order Scale Search for FP4 Quantization
PaperSource excerpt
“Combining Hadamard preprocessing with SoScale is most useful for MXFP4, where it gives the best average score for both Qwen3-8B and Qwen3-14B, and it also gives the best NVFP4 result on Qwen3-14B. However, the best row is not always the Hadamard variant: Qwen3-8B under NVFP4 and Qwen3-14B under HiF4 are best with SoScale alone.” · Show in paper
Verified
Every comparison in the claim is directly supported by the bolded and underlined average scores in Table 2.
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“At the start of each epoch initialize a list with the single entry (u, 0). ... On each raw request r apply these deterministic rules: (i) If every listed tuple covers r, drop the request and leave the list unchanged. ... (ii) Otherwise keep the request. Preserve entries that cover r. Remove every other entry (c, i); if i < M, replace it by the k entries (T(c, r, j), i + 1), for j = 1, . . . , k. An entry with i = M is removed without replacement.” · Show in paper
GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers· Method
PaperCodeSource excerpt
“Our entire GPTQ calibration data consists of 128 random 2048 token segments from the C4 dataset (Raffel et al., 2020) ... Specifically, we always load one Transformer block, consisting of 6 layers, at a time into GPU memory and then accumulate the layer-Hessians and perform quantization. Finally, the current block inputs are sent through the fully quantized block again to produce the new inputs for the quantization of the next block.” · Show in paper
Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention
PaperSource excerpt
“The attention rows compare BF16 SDPA with the NVFP4 Q/K body; QKV and output projections are counted in the complete-step table. Ours is 1.27× faster and uses 51.1%/52.9% less incremental allocation.” · Show in paper
Verified
All quality, error-analysis, and native-hardware speedup/memory claims for Q-Attn match the reported tables and figures.
OpenAI Math GitHub Repository
PaperCodeSource excerpt
“The same asymptotic bounds also suffice for strings with general symbols (assuming exact observation of retained symbols). This concerns sample complexity, not computational efficiency.” · Show in paper
Verified
The preliminary sampling union bound (<= 1/12) and the O(log n) multiplier for the S^n union bound are sound and do not change the asymptotic exponent.
BridgeKV: Large-Model Prefill and Small-Model Decoding for LLM Inference via KV Cache Translation
PaperSource excerpt
“BridgeKV reduces response time from 39.80 s to 9.13 s, a 4.36× speedup over Llama-3.1-Instruct-8B-only inference. Throughput increases from 25.81 tok/s to 113.5 tok/s, a 4.40× gain, because decode follows the Llama-3.2-Instruct-1B anchor. ... Energy per request drops from 11.98 Wh (i.e., watt-hour) for Llama-3.1-Instruct-8B-only inference to 2.78 Wh for BridgeKV, corresponding to a 4.31× reduction.” · Show in paper
Verified
Tables 5 and 6 directly report these exact simulation numbers.
QQQ: Quality Quattuor-Bit Quantization for Large Language Models
PaperCodeSource excerpt
“The per-group QQQ demonstrates competitive performance with with W8A8 SmoothQuant, W4A16 GPTQ, and W4A16 AWQ across various models. For example, per-group QQQ only increases perplexity by up to 0.13 on LLaMA-2-13B compared with them. When benchmarked against W4A4 Atom, both the per-channel and per-group QQQ consistently outperform Atom on all tested models.” · Show in paper
Verified
Table 1 and Table 2 directly substantiate the perplexity and zero-shot accuracy comparisons against SmoothQuant, GPTQ, AWQ, and Atom.
OpenAI Math GitHub Repository
PaperCodeSource excerpt
“The procedure returns Qp(Hz) in the same array, coefficient order, and component format in O(Vd^{λ'}) time, uniformly over all these choices of p, d, r, D, P, S, ρ, w and z. Every returned coefficient belongs to the complex unit disk, and, in the maximum coefficient norm, ∥Qp(Hz) − Hz∥_∞ < √2 2^{-p}.” · Show in paper
Verified
The recurrence unrolling in Section 5.3, guard-width bound in Section 5.4, and uniformity check in Proposition 14 are all rigorously established.
RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
PaperCodeSource excerpt
“As shown in the Figure 7, RocketKV achieves up to 3.7× and 3.3× end-to-end speedups on an A100 and H100 GPU, respectively, as well as up to 32.6% peak memory saving.” · Show in paper
Verified
Figure 7 directly shows the ~3.7x A100 speedup, ~3.3x H100 speedup, and ~32.6% peak memory saving at 100K sequence length.
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“Keep only coarse blocks whose hashes lie in U. Since x + y = 2z, a surviving triple has x = y = z.” · Show in paper
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“The packing permutation, its inverse, and the list of sector starts, widths, and pair counts can all be generated in O(R) operations after the local tables and suffix products have been prepared.” · Show in paper
OpenAI Math GitHub Repository
PaperCodeSource excerpt
“Theorem 12.1 (Fixed inverse-polynomial accuracy). Fix a randomized machine M with polynomial worst-case running time and O(log(n + 2)) work space, and fix an integer d ≥ 1. There is a uniform deterministic transducer that, on every input x of length n, outputs a dyadic rational zx ∈ [0, 1] with |zx − pM(x)| ≤ (n + 2)−d. Its work space and output length are O(log(n + 2)), and its running time is polynomial in n + 2. The constants may depend on M and d, but not on the input.” · Show in paper
Verified
The proof in Section 12.2 accounts for all error terms (hierarchy truncation, statistical error, and digit truncation), space usage (), and running time ().
Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention
PaperSource excerpt
“At d = 4096/6144, a trigger costs 160.00/226.96 ms and amortizes to 0.078/0.111 ms per step, or 0.026%/0.018% of the measured complete step. Added peak allocation is 0.031/0.039 GiB (0.3% in both settings).” · Show in paper
Verified
The arithmetic (160.00/2048 ≈ 0.0781 ms; 226.96/2048 ≈ 0.1108 ms) and percentages against the complete step in Table 5 are consistent.
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“At tier t ∈ {0, . . . , T}, clamp every reference activity to [c−t, ct] and denote the matching law by πt. ... Use q slots at each tier, with product stationary law ν = ⨂Tt=0 π⊗qt .” · Show in paper
M2XFP: A Metadata-Augmented Microscaling Data Format for Efficient Low-bit Quantization· Method
PaperCodeSource excerpt
“M2XFP adopts a hybrid design that assigns: (1) Weights apply Sg-EM-2bit format, enabling fine-grained subgroup-scale refinement through offline adaptive optimization, thereby improving bit efficiency while maintaining fidelity. (2) Activations apply Elem-EM-top1 format, which captures outliers within each subgroup in real time with minimal routing overhead.” · Show in paper
Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention· Method
PaperSource excerpt
“LoRA-SVD parameterizes a training-time weight as W = Wres + βL2L1. It preserves a compact principal subspace and its activation path in BF16 while computing the full-shape residual in NVFP4.” · Show in paper
OpenAI Math GitHub Repository
PaperCodeSource excerpt
“Theorem 11.1 (Realization of the estimator interfaces). Fix the construction constants and an admissible top budget Mtop = O(B). For every actual task tuple reachable from it, the row, vector and prefix interfaces defined in the preceding sections admit deterministic uniform implementations by restoring queries and cursor motions. ... 3. For a root numerical request at clock t, all records retained by the call and its descendants have total length O(Mtop + t + 1), and every descendant clock is O(Mtop + t + 1); the shared facilities use O(B + Mtop + t + 1) space. In particular an actual top request with t = O(B) uses total deterministic O(B) space.” · Show in paper
Verified
The telescoping potential argument, regenerable record design, and contour-based inverse traversal are rigorously proved.