Search publications
GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
PaperCodeSource excerpt
“Specifically, GPTQ can quantize GPT models with 175 billion parameters in approximately four GPU hours, reducing the bitwidth down to 3 or 4 bits per weight, with negligible accuracy degradation relative to the uncompressed baseline. Our method more than doubles the compression gains relative to previously-proposed one-shot quantization methods, preserving accuracy, allowing us for the first time to execute an 175 billion-parameter model inside a single GPU for generative inference. ... We show experimentally that these improvements can be leveraged for end-to-end inference speedups over FP16, of around 3.25x when using high-end GPUs (NVIDIA A100) and 4.5x when using more cost-effective ones (NVIDIA A6000).” · Show in paper
Verified
The experimental results in Tables 2, 5, and 6 directly support the 4-hour quantization time, 3-4 bit accuracy preservation, single-GPU execution on A100, and 3.24x/4.53x speedups.
Towards Load-Aware Prefill Deflection for Disaggregated LLM Serving· Method
PaperCodeSource excerpt
“KAIROS is implemented on top of vLLM v0.18.1 [8] which supports disaggregated prefill/decode through a KV transfer API. KAIROS adds approximately 2000 lines of new code across the profiling for step latency models, the Deflection Dispatcher (which contains the analytical models and the deflection algorithm), the modifications to the scheduler on the decode nodes and plugins to report node state to the Deflection Dispatcher.” · Show in paper
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“A key is a grid-valued function on the bit cube, specified by its ordered list of essential bit names and its truth table.” · Show in paper
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“At each stage we replace the old description instead of carrying its history into the new message.” · Show in paper
How Small Can You Go? A Controlled Study of LoRA Rank, Target Modules, and Quantization Trade-offs for Text-to-SQL on a 60M-Parameter Model· Method
PaperCodeSource excerpt
“Table 1 lists all evaluated configurations; each changes one variable relative to the LoRA r=8, {q,v}, FP16 reference.” · Show in paper
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“For a positive absolute σ < 1, to be chosen below, set bi = σvi, ηi = 1 / (1 + √(1 − ∥bi∥2 2)), Ci = I − ηibibT i .” · Show in paper
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“For X \sim \mu, form the Gaussian observation h_p = B_p X + \beta_p, Cov(\beta_p, \beta_q) = . (3.1)” · Show in paper
Likelihood truncation on regular dyadic cells with finite-horizon success bounding.
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“Lemma 7.4 (A regular-cell likelihood cutoff). There are absolute a, c1 > 0 and a dimension threshold such that the following holds. Let P be regular in a cell C, let A ∼ γn,d be independent of z ∼ P, and let W = f(A, As(z)) for a Borel map f with at most N values.” · Show in paper
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“The local orbit includes the higher active determining slots, so its replacement must retain the formal identity π^0 g = PY from Lemma 7.5. This identity will restore the old integer arguments exactly in Section 9.” · Show in paper
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“Retaining the completed target while reversing its auxiliary computation follows the reversible-cleanup principle of Bennett [4, pp. 525–526]. The two-buffer schedule, local transitions, and certification guarantees needed here are established in the following proof.” · Show in paper
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“Choose an integer m and independent uniform binary words of length m, one for each letter, and put u = floor(m/8). We seek an absolute delta > 0 such that every pair of length-u intervals from differently labelled codewords has longest common subsequence at most (1 - delta)u.” · Show in paper
BridgeKV: Large-Model Prefill and Small-Model Decoding for LLM Inference via KV Cache Translation· Method
PaperSource excerpt
“The first stage trains only ψ, with both anchors frozen, and combines two losses: an alignment loss that keeps the stitched cache close to the native cache of Md, and a distillation loss that makes Md, reading the stitched cache, match the next-token distribution of Mp.” · Show in paper
SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models· Method
PaperCodeSource excerpt
“Y = (Xdiag(s) −1 ) · (diag(s)W) = XˆWˆ (3)” · Show in paper
QQQ: Quality Quattuor-Bit Quantization for Large Language Models
PaperCodeSource excerpt
“Our extensive experiments show that QQQ achieves performance on par with existing state-of-the-art LLM quantization methods while significantly accelerating inference, achieving speed boosts up to 2.24 ×, 2.10×, and 1.25× compared to FP16, W8A8, and W4A16, respectively.” · Show in paper
Verified
The reported perplexity, zero-shot accuracy, and vLLM throughput benchmarks on LLaMA models support the claimed parity with SOTA quantization methods and the specific speedup figures.
Learning New Facts with QLoRA: An Acquisition-Retention Frontier
PaperCodeSource excerpt
“Comparing full fine-tuning (FFT) with quantized low-rank adaptation (QLoRA) at ranks 8, 16, 32, and 64, we find that rank induces a clear acquisition–retention frontier. Low-rank QLoRA preserves out-of-domain (OOD) performance but acquires fewer facts, whereas higher ranks improve same-fact paraphrase generalization at an increasing cost in performance on unrelated benchmarks. FFT behaves as a conservative baseline: it retains general capabilities well, but does not reach the highest factual-acquisition regime.” · Show in paper
Verified
The five-seed results in Figure 1 and Table 5 directly support the full claim, including the relative positions of FFT and QLoRA ranks 8, 16, 32, and 64.
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“alpha = log 3, alpha_m = floor(log(3^m))/m, e = floor(g(1/2 + alpha_m(k - 1/4))).” · Show in paper
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“For a nonraw row v of width r, minimize over all original popular centres (D, r0) in the block the cost b + 10a + 10c+ + (−c)+, b = − log(Dr0), a = log v (v, D) , c = log r r0 , (3.3)” · Show in paper
OpenAI Math GitHub Repository
PaperCodeSource excerpt
“For all sufficiently large d, set k = 2\lfloor d/16\rfloor, m = k, t = m + 1, \ell = 4k. Suppose p = f\sigma_d is a probability with 0 \le f \le L < \infty. In the block experiment above, let A have k rows and let B have \ell rows. There is an absolute constant C such that I(S; W \mid B, BS) \le H(W)/t + Cd + C\log(2 + \log L).” · Show in paper
Verified
All integrability conditions, null-set transfers, and cancellations are rigorously verified in Sections 3 and 4.
OpenAI Math GitHub Repository
PaperCodeSource excerpt
“If H(W) ≤ d^2, then, for an absolute constant K, I0(S; W | G, GS) ≤ I1(S; W | G, GS) + Kd.” · Show in paper
Verified
All dimensional conditions (, ), measurability, integrability (Lemma 2.3), and entropy bounds are rigorously checked.
OpenAI Math GitHub Repository· Method
PaperCodeSource excerpt
“Lemma 6.1 (Bounded-density split). Let µ be a probability supported on a dyadic cell A, with bounded density relative to τA. There is a finite measurable partition into positive-probability labels i, with probabilities pi, descendant cells Ai of relative depths ji, and numbers qi ∈ (0, 1].” · Show in paper