Headline finding
GPTQ is a one-shot post-training weight quantization method based on approximate second-order information that can quantize 175-billion-parameter GPT/OPT models in approximately four GPU hours down to 3 or 4 bits per weight with negligible accuracy degradation, more than doubling compression gains relative to prior one-shot methods and enabling single-GPU generative inference with end-to-end speedups of around 3.25x on NVIDIA A100 and 4.5x on NVIDIA A6000 GPUs.
The experimental results in Tables 2, 5, and 6 directly support the 4-hour quantization time, 3-4 bit accuracy preservation, single-GPU execution on A100, and 3.24x/4.53x speedups.
“Specifically, GPTQ can quantize GPT models with 175 billion parameters in approximately four GPU hours, reducing the bitwidth down to 3 or 4 bits per weight, with negligible accuracy degradation relative to the uncompressed baseline. Our method more than doubles the compression gains relative to previously-proposed one-shot quantization methods, preserving accuracy, allowing us for the first time to execute an 175 billion-parameter model inside a single GPU for generative inference. ... We show experimentally that these improvements can be leveraged for end-to-end inference speedups over FP16, of around 3.25x when using high-end GPUs (NVIDIA A100) and 4.5x when using more cost-effective ones (NVIDIA A6000).” · Show in paper