Headline finding
SmoothQuant is a training-free, accuracy-preserving, and general-purpose post-training quantization (PTQ) solution that enables INT8 quantization of both weights and activations (W8A8) for all matrix multiplications in LLMs—including OPT, BLOOM, GLM, MT-NLG, Llama-1/2, Falcon, Mistral, and Mixtral models—achieving up to 1.56× speedup and 2× memory reduction with negligible loss in accuracy, and enabling serving a 530B LLM within a single node.
The experimental results across accuracy benchmarks (Tables 3–7, 9) and system benchmarks (Figures 8–9, Tables 8, 10) directly support the claimed speedup, memory reduction, model coverage, and single-node 530B serving.
“We propose SmoothQuant, a training-free, accuracy-preserving, and general-purpose post-training quantization (PTQ) solution to enable 8-bit weight, 8-bit activation (W8A8) quantization for LLMs. Based on the fact that weights are easy to quantize while activations are not, SmoothQuant smooths the activation outliers by offline migrating the quantization difficulty from activations to weights with a mathematically equivalent transformation. SmoothQuant enables an INT8 quantization of both weights and activations for all the matrix multiplications in LLMs, including OPT, BLOOM, GLM, MT-NLG, Llama-1/2, Falcon, Mistral, and Mixtral models. We demonstrate up to 1.56× speedup and 2× memory reduction for LLMs with negligible loss in accuracy. SmoothQuant enables serving 530B LLM within a single node.” · Show in paper