By Elias Frantar, Saleh Ashkboos, Torsten Hoefler, Dan AlistarhSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2210-17323.git
Cite
Version link
https://openversions.com/index/arxiv-2210-17323/commit/5614170ba5aefaf7f37bca05a95ba02f5b7d0626
Version identifier (commit hash)
5614170ba5aefaf7f37bca05a95ba02f5b7d0626

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Latest version

Checks complete

Archive from arXiv Source: https://arxiv.org/abs/2210.17323v2 OpenVersions: 8f141526-9de0-4c2a-9c76-ad23cfce24f7

Posted

Claims

Findings

Headline finding

GPTQ is a one-shot post-training weight quantization method based on approximate second-order information that can quantize 175-billion-parameter GPT/OPT models in approximately four GPU hours down to 3 or 4 bits per weight with negligible accuracy degradation, more than doubling compression gains relative to prior one-shot methods and enabling single-GPU generative inference with end-to-end speedups of around 3.25x on NVIDIA A100 and 4.5x on NVIDIA A6000 GPUs.

GPTQ quantizes 1-3 billion parameter models in a matter of minutes and 175B-parameter models (OPT-175B and BLOOM-176B) in approximately 4 GPU hours on a single NVIDIA A100 GPU.

Quantizing the weights of all rows in the same fixed order yields final squared error similar to OBQ's greedy order on large models, reducing overall runtime from O(drow⋅dcol3)O(d_{\text{row}} \cdot d_{\text{col}}^3) to O(max⁡{drow⋅dcol2,dcol3})O(\max\{d_{\text{row}} \cdot d_{\text{col}}^2, d_{\text{col}}^3\}).

Lazy batch-updates with blocksize B=128B = 128 address the memory-throughput bottleneck, providing an order of magnitude speedup in practice, while Cholesky reformulation with mild dampening (λ=1%\lambda = 1\% of the average diagonal) prevents numerical indefiniteness of HF−1\mathbf{H}_F^{-1} on models with billions of parameters.

At 4-bit precision, GPTQ models reach at most ≤0.25\le 0.25 higher perplexity than full-precision versions on OPT-175B and BLOOM-176B across WikiText2, PTB, and C4; at 3-bit precision, RTN collapses while GPTQ maintains good performance, losing only 0.3−0.60.3 - 0.6 perplexity points for more than 5×5\times compression.

Using bespoke quantized-matrix full-precision-vector product kernels, 3-bit OPT-175B fits into a single 80GB NVIDIA A100 GPU (reducing GPUs from 5 to 1) and achieves a 3.24×3.24\times per-token latency speedup over FP16 on A100 and a 4.53×4.53\times speedup on 2×2\times NVIDIA A6000 GPUs (reducing GPUs from 8 to 2).

With finer-granularity grouping, GPTQ achieves reasonable accuracy in the extreme quantization regime (~2.2 bits with group-size 128, ~2.6 bits with group-size 32, and ternary (-1, 0, +1) with group-size 8 achieving 9.20 WikiText2 perplexity on OPT-175B).

Methods

Layer-wise post-training quantization objective minimizing squared output error argminW^∥WX−W^X∥22\text{argmin}_{\widehat{\mathbf{W}}} \|\mathbf{W}\mathbf{X} - \widehat{\mathbf{W}}\mathbf{X}\|_2^2 on a fixed quantization grid.

Source excerpts

“Then, the objective is to find a matrix of quantized weights Wc which minimizes the squared error, relative to the full precision layer output. Formally, this can be restated as argminWc ||WX − WX c ||2_2.” · Show in paper

GPTQ algorithm (Algorithm 1): quantizing all rows of W\mathbf{W} in the same column order, using lazy batch-updates with blocksize B=128B = 128, and precomputing inverse Hessian information via Cholesky decomposition of H−1=(2XX⊤+λI)−1\mathbf{H}^{-1} = (2\mathbf{X}\mathbf{X}^\top + \lambda\mathbf{I})^{-1} with dampening λ\lambda set to 1%1\% of the average diagonal value.

Source excerpts

“Concretely, we apply the algorithm to B = 128 columns at a time, keeping updates contained to those columns and the corresponding B × B block of H−1 ... For smaller models, applying dampening, that is adding a small constant λ (we always choose 1% of the average diagonal value) to the diagonal elements of H ... Algorithm 1 Quantize W given inverse Hessian H−1 = (2XX> + λI)−1 and blocksize B.” · Show in paper

Zero-shot calibration using 128 random 2048-token segments from the C4 dataset, loading one Transformer block (6 layers) at a time into GPU memory and propagating quantized block outputs as inputs to the next block.

Source excerpts

“Our entire GPTQ calibration data consists of 128 random 2048 token segments from the C4 dataset (Raffel et al., 2020) ... Specifically, we always load one Transformer block, consisting of 6 layers, at a time into GPU memory and then accumulate the layer-Hessians and perform quantization. Finally, the current block inputs are sent through the fully quantized block again to produce the new inputs for the quantization of the next block.” · Show in paper

Quantized-matrix full-precision-vector product GPU kernel that dynamically dequantizes weights during matrix-vector multiplication for batch-size-1 generative inference without activation quantization.

Source excerpts

“We address this problem by developing a quantized-matrix full-precision-vector product kernel which performs a matrix vector product by dynamically dequantizing weights when needed. Most notably, this does not require any activation quantization.” · Show in paper