Search publications
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“Divide the p levels into consecutive blocks of B levels each. At the first level of every block after the first, we reset the selection cell to the whole space. Older retained references are kept unless they meet the conflict rule specified below. Independently of the disorder, banks and path, draw independent thresholds r_i, uniform on I_i = [t_0 + 4(i-1)\rho, t_0 + (4(i-1)+1)\rho].” · Show in paper
BridgeKV: Large-Model Prefill and Small-Model Decoding for LLM Inference via KV Cache Translation
PaperSource excerpt
“The batched implementation adds 1.6–6.7% to the prefill time of the prefill anchor across the four pairs, whereas the per-layer implementation adds 5–51% (Figure 4b). ... The request takes 2.96 s, close to the 2.80 s of Llama-3.2-Instruct-1B alone and 3.0× faster than the 8.83 s of Llama-3.1-Instruct-8B alone (Figure 1).” · Show in paper
Verified
Measured median latencies on one A40 GPU in Figure 1, Figure 4, and Section 4.3 directly support the reported percentages and speedups.
SoScale: Second-Order Scale Search for FP4 Quantization· Method
PaperSource excerpt
“Following GPTQ, we compute an upper-triangular factor U of the damped inverse Hessian, U⊤U = (H + λI)−1, (9) ... The block error is then propagated to the future blocks through the back-substitution: Wf:,F ← Wf:,F − EQU−1Q,QUQ,F. (13)” · Show in paper
OpenAI Math GitHub Repository
PaperCodeSource excerpt
“Proposition 7.1 (Embedded comparator cost). There are finite alphabets A_1, ..., A_J, depending only on Y, k and the fixed constants, and online random maps ξ_t : Y → A_1 × · · · × A_J with the following properties. Each coordinate cell at level j has diameter at most 40r_j and an anchor within 20r_j of all its points. Thus ξ_t is injective. Give the full alphabet tree the leaf distance when the first differing coordinate is j. For the selected hidden members X_t^1, ..., X_t^k, P := E ∑_t ∑_{a=1}^k d_T((^a), ξ_t(X_t^a)) ≤ CℓQ + . (7.2) Here C is absolute and < ∞ is independent of the input law and its horizon.” · Show in paper
Verified
The proof in Sections 6 and 7 carefully separates conditioning on the filtration for pilot potentials from averaging over the partition tape for separation and retirement probabilities.
Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction
PaperCodeSource excerpt
“As shown in Table 5, using the token contextual embedding xt as input performs better, particularly under tight budgets, suggesting that xt provides a stronger long-horizon utility signal than the projected KV representations alone.” · Show in paper
Verified
Table 5 directly supports the claim.
Prefix-cost-predicted alignment bands with velocity tests and rounded canonical actions.
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“Let Cb,r be the following set of shared-endpoint actions on the scale-b grid. The outside endpoints are exactly r, consecutive child endpoints agree, all endpoint shifts are within 6b of p0 = r− − l, and the shift p at every internal cut satisfies |p − p0 − vbΦj(p)| ≤ τθb/M.” · Show in paper
QLoRA: Efficient Finetuning of Quantized LLMs
PaperCodeSource excerpt
“(1) 4-bit NormalFloat, an information theoretically optimal quantization data type for normally distributed data that yields better empirical results than 4-bit Integers and 4-bit Floats.” · Show in paper
Verified
Empirical results across perplexity (Table 2), zero-shot accuracy (Figure 3), and MMLU (Table 4) consistently support that NF4 outperforms FP4 and Int4.
SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference· Method
PaperCodeSource excerpt
“we compute during prefill and mark positions with as boundaries, where and are the sequence mean and standard deviation, and boundaries closer than tokens are merged, keeping the highest-surprisal one.” · Show in paper
MomentKV: Closing the Directional Gap in KV Cache Eviction for Long-Context Inference· Method
PaperSource excerpt
“This yields the lower bound \hat{Z}_{\mathcal{E}} = n_e \cdot \exp(q^\top \bar{k}/\sqrt{d}) \le Z_{\mathcal{E}}, with relative error O(\sigma^2) controlled by the logit variance (see Appendix B). With both components estimated, the corrected output takes the form \hat{f}(q) = \hat{w}_{\mathcal{R}} f_{\mathcal{R}}(q) + (1 - \hat{w}_{\mathcal{R}}) \hat{f}_{\mathcal{E}}(q), \hat{w}_{\mathcal{R}} = Z_{\mathcal{R}} / (Z_{\mathcal{R}} + \hat{Z}_{\mathcal{E}}).” · Show in paper
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“For a prescribed center path c(σ) and a spatial scale δ > 0, the curl field V(σ, x) = ∇x × [ζ((x − c(σ))/δ) (c˙(σ) × (x − c(σ)))/2] is divergence-free and equals c˙(σ) on a neighborhood of the moving center.” · Show in paper
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“Theorem 5.2 (One-step coverage transfer).” · Show in paper
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“Let Ds > 0, and let Q satisfy (7.1), with ∥v∥2 ≤ Ds. Put n = r/(2Ds), P = (Im − vv^T/(4D_s^2))^{1/2}. For a deterministic x ∈ R^d, let Z, G1, . . . , Gm be independent standard Gaussian vectors in R^d. Then the output of Q(x + nZ; (P ⊗ Id)G) has exactly the law of Q(x; G).” · Show in paper
ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMs· Method
PaperCodeSource excerpt
“Online Activation Quantization. We process X via: (1) Reordering and Primary Quantization. X is reordered, followed by block-wise quantization QX = round(X/sX). (2) Residual Compensation. We isolate outliers Xo, compute residuals Ro = Xo − sXo · QXo , and quantize them into QRo . (3) Augmentation. We concatenate along dimension K as QXaug = [QX | QRo ] with combined scales sXaug = [sX | sRo ].” · Show in paper
End-to-End Context Compression at Scale· Method
PaperCodeSource excerpt
“After the first 768 generated tokens, we encode the oldest 512 generated tokens into latent memory tokens at a 16x compression ratio, resulting in 32 latent tokens, and keep the most recent 256 tokens uncompressed. Generation then continues from the compacted context and latest raw tokens, and the procedure repeats after every further 512 generated tokens.” · Show in paper
A JoLT for the KV cache: Near-Lossless KV Cache Compression via Joint Rank-bit Allocation· Method
PaperCodeSource excerpt
“FlashJoLT replaces this SVD with a randomized approximation [11] that computes only the leading qcap token-mode singular vectors. The feature-mode SVD is kept exact because dh is small (128), and the rest of the pipeline, including the Lagrangian allocator, is unchanged.” · Show in paper
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“We use a local-lemma argument [3, Section 2, pp. 616–617], recalling the required finite probability estimate with its proof.” · Show in paper
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“If it overflows, discard the task's workspace and replace the task by two right tasks that fix the next bit of its M_0 mask to zero and one, respectively. Process these tasks depth first, in a fixed order of the top-mixer bits.” · Show in paper
Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention· Method
PaperSource excerpt
“Table 2 assigns NVFP4 to the QK⊤ and dS paths and retains the softmax-sensitive P V , P⊤dO, and dOV ⊤ products in BF16. Quantized views are reused between forward and backward to avoid precision mismatch and repeated memory traffic.” · Show in paper
OpenAI Math GitHub Repository
PaperCodeSource excerpt
“Theorem 4.1 (Two binary substitutions). Let A be a finite alphabet with A >= 2 letters. Each of the two constructions below gives an integer width w >= 1 with w = O(log(2A)), an absolute constant a > 0, and a map c : A -> {0, 1}^w such that, simultaneously for every integer L >= 0 and all x, y in A^L, a w r_L(x, y) <= (c(x), c(y)) <= w r_L(x, y). (23) Here c extends to words by concatenation. For arbitrary finite words, ED(c(x), c(y)) <= w ED(x, y). (24)” · Show in paper
Verified
Both probabilistic existence bounds and deterministic matching recovery arguments are proved without gaps.
GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers· Method
PaperCodeSource excerpt
“We address this problem by developing a quantized-matrix full-precision-vector product kernel which performs a matrix vector product by dynamically dequantizing weights when needed. Most notably, this does not require any activation quantization.” · Show in paper