By Ying Zhang, Peng Zhang, Mincong Huang, Jingyang Xiang, Yujie Wang, Chao Wang, Yineng Zhang, Lei Yu, Chuan Liu, Wei LinSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2406-09904.git
Cite
Version link
https://openversions.com/index/arxiv-2406-09904/commit/6356107511bc479a552e6957da5ce94e20267371
Version identifier (commit hash)
6356107511bc479a552e6957da5ce94e20267371

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Claims

Findings

Headline finding

QQQ, a Quality Quattuor-bit Quantization method with 4-bit weights and 8-bit activations, achieves performance on par with existing state-of-the-art LLM quantization methods while significantly accelerating inference, achieving speed boosts up to 2.24x, 2.10x, and 1.25x compared to FP16, W8A8, and W4A16, respectively.

Per-group QQQ demonstrates competitive WikiText2 perplexity with W8A8 SmoothQuant, W4A16 GPTQ, and W4A16 AWQ across LLaMA models (increasing perplexity by up to 0.13 on LLaMA-2-13B), and both per-channel and per-group QQQ consistently outperform W4A4 Atom on all tested models.

Integrated into vLLM, QQQ outperforms FP16, SmoothQuant, AWQ, and Marlin across a range of batch sizes in most cases, achieving up to 2.24x, 2.10x, 1.59x, and 1.25x faster throughput on LLaMA-2-13B, respectively.

The specialized per-channel W4A8 GEMM and per-group W4A8 GEMM achieve speedups of up to 3.67x and 3.29x over PyTorch FP16 GEMM, respectively, and W4A8 GEMMs achieve up to 2.51x and 1.71x faster performance compared to W8A8 GEMM and W4A16 Marlin GEMM.

Combining adaptive smoothing with GPTQ yields the lowest perplexity scores compared to the vanilla W4A8 baseline and adaptive smoothing alone, reducing perplexity by 0.47 in per-group quantization on LLaMA-3-8B.

The FastFP16toINT8 conversion technique accelerates per-group W4A8 GEMM by up to 2.02x compared to using the standard FP16 to INT8 conversion instruction.

Methods

Adaptive smoothing selectively smooths activation channels with outliers exceeding a grid-searched threshold σ\sigma (T={t∣max⁡(abs(X:,t))<σ}\mathcal{T} = \{ t \mid \max(\text{abs}(\mathbf{X}_{:,t})) < \sigma \}) and optimizes smoothing parameters s\mathbf{s} to minimize ∥Q(X:,T⊘s)Q(WT,:⊙s)−X:,TWT,:∥22\| Q(\mathbf{X}_{:,\mathcal{T}} \oslash \mathbf{s}) Q(\mathbf{W}_{\mathcal{T},:} \odot \mathbf{s}) - \mathbf{X}_{:,\mathcal{T}} \mathbf{W}_{\mathcal{T},:} \|_2^2.

Source excerpts

“Specifically, we introduce an outlier threshold σ, which is determined through a grid search within the range [0, max(abs(X))]. Activation channels that exhibit outliers exceeding this threshold σ are then selectively smoothed:” · Show in paper

Hessian-based quantization compensation adopts the layer-wise GPTQ framework, iteratively quantizing weight rows Wi\mathbf{W}_i and adjusting the remaining full-precision weights FF by δF=−Wi−Q(Wi)[HF−1]ii⋅(HF−1):,i\boldsymbol{\delta}_F = -\frac{\mathbf{W}_i - Q(\mathbf{W}_i)}{[\mathbf{H}_F^{-1}]_{ii}} \cdot (\mathbf{H}_F^{-1})_{:,i} where HF=2XFTXF\mathbf{H}_F = 2\mathbf{X}_F^T \mathbf{X}_F.

Source excerpts

“We adopt a layer-wise quantization framework GPTQ (Frantar et al., 2022), which advocates for a precise and efficient Hessian-based quantization compensation technique.” · Show in paper

Per-channel W4A8 GEMM converts INT4 weights to INT8 using FastINT4toINT8 (multiplying by 16 / left-shifting by 4 bits), executes INT8 GEMM, and dequantizes the INT32 output to FP16 while folding the factor-of-16 scaling offline into the weight quantization scale.

Source excerpts

“Our method involves an initial conversion of INT4 weights to INT8 before executing the INT8 GEMM, followed by a dequantization step that transforms the INT32 output of the INT8 GEMM into a FP16 result. To ensure high efficiency in the type conversion, we utilize the FastINT4toINT8 (Li et al., 2023) technique.” · Show in paper

Per-group W4A8 GEMM converts INT4 weights to FP16 via FastINT4toFP16, applies FusedDequantQuant (merging per-group dequantization and per-channel re-quantization scales), converts FP16 to INT8 via FastFP16toINT8 (using FMA with magic number 128 + 1024, PRMT, and XOR with 0x80), executes INT8 GEMM, and dequantizes INT32 output to FP16.

Source excerpts

“Initially, INT4 weights are converted to FP16 using the FastINT4toFP16 (Kim et al., 2022) method. Subsequently, each group’s weight is dequantized by multiplying it by its respective scale. To re-quantize into INT8, the result is divided by the per-channel quantization scale. These two steps are elegantly merged into a single operation, FusedDequantQuant, by dividing the per-group scale by the per-channel scale, as illustrated in Figure 4.” · Show in paper

“In practice, the fusion of the per-group quantization scale multiplication with the addition of the magic number 128 + 1024 is executed through a single FMA (Fused Multiply-Add) instruction. Furthermore, the process leverages two expedient bit manipulation instructions, PRMT and XOR, to finalize the conversion.” · Show in paper

Experimental evaluation uses LLaMA-1/2/3 models calibrated on 128 sequences from the PILE dataset, evaluated on WikiText2 perplexity and five zero-shot tasks (PIQA, HellaSwag, WinoGrande, ARC-e, ARC-c) via lm_eval, and benchmarked on NVIDIA A100 80G GPUs using vLLM v0.4.1 with PyTorch 2.1.2 and CUDA 11.8.

Source excerpts

“We conduct the experiments using the LLaMA series (Touvron et al., 2023a,b). We evaluate the model performance on language modeling and zero-shot tasks. For the language modeling aspect, we utilize the WikiText2 (Merity et al., 2016) dataset to measure perplexity. For the zero-shot tasks, we extend the evaluation to a variety of datasets, including PIQA (Bisk et al., 2020), HellaSwag (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2021), and ARC (Clark et al., 2018), utilizing the lm_eval (Gao et al., 2023) toolkit for a comprehensive analysis. Additionally, we select a subset of 128 sequences from the PILE (Gao et al., 2020) dataset to facilitate calibration.” · Show in paper

“The inference system that we have develop to support QQQ is based on vLLM1 (Kwon et al., 2023) v0.4.1. The system’s capabilities are further enhanced by the implementation of W4A8 GEMM kernels, which are based on the advanced Marlin2 (Frantar and Alistarh, 2024) kernels. All experimental procedures are conducted on NVIDIA A100 80G GPUs under PyTorch 2.1.2 with CUDA 11.8.” · Show in paper