By Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, Song HanSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2211-10438.git
Cite
Version link
https://openversions.com/index/arxiv-2211-10438/commit/967662418d8ce0e7695cefe6bad5e9ba2b5c52be
Version identifier (commit hash)
967662418d8ce0e7695cefe6bad5e9ba2b5c52be

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Latest version

Checks complete

Archive from arXiv Source: https://arxiv.org/abs/2211.10438v7 OpenVersions: 58dadef3-f01f-4f43-ab1a-6e92187c09e7

Posted

Claims

Findings

Headline finding

SmoothQuant is a training-free, accuracy-preserving, and general-purpose post-training quantization (PTQ) solution that enables INT8 quantization of both weights and activations (W8A8) for all matrix multiplications in LLMs—including OPT, BLOOM, GLM, MT-NLG, Llama-1/2, Falcon, Mistral, and Mixtral models—achieving up to 1.56× speedup and 2× memory reduction with negligible loss in accuracy, and enabling serving a 530B LLM within a single node.

SmoothQuant maintains the FP16 accuracy of OPT-175B after INT8 quantization across all 7 zero-shot evaluation datasets and WikiText-2 under all three quantization schemes (O1, O2, O3), whereas W8A8, ZeroQuant, and Outlier Suppression baselines produce nearly random results.

SmoothQuant can quantize diverse LLMs—including all existing open LLMs beyond 100B parameters (OPT-175B, BLOOM-176B, GLM-130B), instruction-tuned LLMs (OPT-IML-30B), and newer architectures (LLaMA, Llama-2, Falcon, Mistral, Mixtral)—into INT8 with negligible accuracy or perplexity degradation.

SmoothQuant-O3 achieves up to 1.51× speedup and 1.96× memory saving on PyTorch (single NVIDIA A100-80GB GPU) and up to 1.56× speedup and nearly 2× memory reduction on FasterTransformer compared to FP16, while enabling multi-GPU models (OPT-66B, OPT-175B) to run at similar or better latency using half the number of GPUs and accelerating autoregressive decoding by up to 1.42×.

SmoothQuant enables W8A8 quantization of MT-NLG 530B with negligible accuracy loss, allowing the 530B model to be served within a single 8×A100 80GB GPU node using half the number of GPUs (8 instead of 16) at a similar latency.

Simulated per-channel activation quantization bridges the accuracy gap with the FP16 baseline, but per-channel activation quantization does not map well to hardware-accelerated GEMM kernels.

Choosing the migration strength α\alpha from the sweet spot region (0.4–0.6 for OPT-175B) yields small quantization errors for both weights and activations and maintains model performance after quantization.

Methods

SmoothQuant migrates quantization difficulty from activations to weights using a mathematically equivalent per-channel scaling transformation Y=(Xdiag(s)−1)⋅(diag(s)W)=X^W^\mathbf{Y} = (\mathbf{X}\text{diag}(\mathbf{s})^{-1}) \cdot (\text{diag}(\mathbf{s})\mathbf{W}) = \hat{\mathbf{X}}\hat{\mathbf{W}}, where sj=max⁡(∣Xj∣)α/max⁡(∣Wj∣)1−αs_j = \max(|\mathbf{X}_j|)^\alpha / \max(|\mathbf{W}_j|)^{1-\alpha}, and fuses the smoothing factor offline into previous layers' parameters.

Source excerpts

“Y = (Xdiag(s) −1 ) · (diag(s)W) = XˆWˆ (3)” · Show in paper

“sj = max(|Xj |) α / max(|Wj |) 1−α (4)” · Show in paper

In Transformer blocks, SmoothQuant quantizes the inputs and weights of compute-heavy operators (linear layers and batched matrix multiplication BMM in attention) to INT8, while keeping activations in FP16 for lightweight element-wise operations such as ReLU, Softmax, and LayerNorm.

Source excerpts

“We quantize the inputs and weights of compute-heavy operators like linear layers and BMM in attention layers with INT8, while keeping the activation as FP16 for other lightweight element-wise operations like ReLU, Softmax, and LayerNorm.” · Show in paper

Smoothing factors and static quantization step sizes are calibrated once offline using 512 random sentences from the Pile pre-training dataset, and the migration strength α\alpha is selected via a quick grid search on a subset of the Pile validation set.

Source excerpts

“We get a suitable α by running a quick grid search on a subset of the Pile (Gao et al., 2020) validation set. To get the statistics of activations, we calibrate the smoothing factors and the static quantization step sizes once with 512 random sentences from the pre-training dataset Pile, and apply the same smoothed and quantized model for all downstream tasks.” · Show in paper

SmoothQuant is implemented in both PyTorch Huggingface and FasterTransformer by replacing FP16 linear modules and BMM functions with INT8 linear modules and BMM functions built using CUTLASS INT8 GEMM kernels.

Source excerpts

“In both PyTorch Huggingface and FasterTransformer frameworks, we implement INT8 linear modules and the batched matrix multiplication (BMM) function with CUTLASS INT8 GEMM kernels. We simply replace the original floating point (FP16) linear modules and the bmm function with our INT8 kernels as the INT8 model.” · Show in paper