By Haoqian Meng, Yilun Luo, Yafei Zhao, Wenyuan Liu, Peng Zhang, Xindian MaSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2601-07475.git
Cite
Version link
https://openversions.com/index/arxiv-2601-07475/commit/225fc0a8f3153afa87e8aca10233460795de1506
Version identifier (commit hash)
225fc0a8f3153afa87e8aca10233460795de1506

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Claims

Findings

Headline finding

ARCQuant boosts NVFP4 quantization performance for LLMs via Augmented Residual Channels, maintaining a strictly unified NVFP4 format to achieve state-of-the-art accuracy comparable to full-precision baselines in perplexity and downstream tasks, while achieving up to 3x speedup over FP16 on RTX 5090 and RTX PRO 6000 GPUs.

For compensated outlier channels, ARCQuant's dual-stage NVFP4 quantization mechanism achieves a worst-case error bound comparable to that of standard single-stage MXFP8.

ARCQuant consistently achieves the best results among W4A4 methods on Llama 3.1-8B, Qwen2.5-7B, and Qwen2.5-32B, outperforming Atom and FlatQuant, surpassing the W4A8 RTN baseline in perplexity and MMLU on Llama 3.1-8B and Qwen2.5-7B, and matching FP16 on Qwen2.5-32B within a negligible margin.

On NVFP4, conventional PTQ methods such as QuaRot and SmoothQuant yield diminishing returns or performance regression compared to RTN, whereas ARCQuant consistently achieves the best results across all metrics.

ARCQuant incurs only 3%–9% latency overhead compared to uncompensated NVFP4 while achieving 2.0x–2.5x prefill speedup on RTX PRO 6000 (Qwen2.5-7B), 3.5x prefill speedup on RTX 5090 (Llama 3.1-8B), and 1.5x–2.8x memory reduction compared to FP16.

Integrated into vLLM on an RTX 5090 (Qwen2.5-7B, batch size 8, generation length 128), ARCQuant achieves a 1.96x decoding speedup over FP16 at sequence length 1024 and a 2.08x decoding speedup over FP16 at sequence length 2048, substantially outperforming FP8.

ARCQuant generalizes to INT4 and MXFP4 formats, outperforming RTN baselines on Llama 3.1-8B (e.g., reducing INT4 perplexity by 0.89 and improving zero-shot accuracy by 2.12%).

ARCQuant is robust to calibration data selection, showing < 0.03 PPL and < 0.03% zero-shot accuracy fluctuations across WikiText2, C4, and HumanEval on Llama 3.1-8B, and retaining over 99% of FP16 accuracy on coding and math tasks when calibrated on WikiText2.

ARCQuant scales effectively to larger dense models (Llama 3.1-70B) and sparse MoE architectures (Mixtral 8x7B-Instruct), consistently improving over the NVFP4+RTN baseline.

Methods

Adaptive Outlier Identification: Using calibration data, channels are reordered by their absolute maximums, and the top-SS channels exceeding the layer-wise threshold τ=2−3M\tau = 2^{-3} M (reflecting the 3-bit exponent difference between E5M2 and E2M1) are selected for residual compensation.

Source excerpts

“Utilizing calibration data, we pre-determine both the channel reordering indices and the number of outlier channels S. We first reorder channels based on their absolute maximums, adopting the sorting strategy from Atom (Zhao et al., 2024). We then determine the layer-wise maximum M, which defines the dynamic range of a per-tensor FP8 (E5M2) reference. We set the selection threshold τ = 2−3M. This reflects the 3-bit difference in exponent width between the reference E5M2 (5 bits) and the target E2M1 (2 bits).” · Show in paper

Dual-Stage Quantization with Augmented Residual Channels: Activations are reordered, quantized to QX=round(X/sX)Q_X = \text{round}(X/s_X), and augmented along the reduction dimension with quantized residuals QRoQ_{R_o} of the outlier channels (QXaug=[QX∣QRo]Q_{X_{\text{aug}}} = [Q_X \mid Q_{R_o}]), while weights are reordered offline and duplicated for outlier channels (QWaug=[QW∣QWo]Q_{W_{\text{aug}}} = [Q_W \mid Q_{W_o}]) so a single unified-precision NVFP4 GEMM computes both primary and compensation terms.

Source excerpts

“Online Activation Quantization. We process X via: (1) Reordering and Primary Quantization. X is reordered, followed by block-wise quantization QX = round(X/sX). (2) Residual Compensation. We isolate outliers Xo, compute residuals Ro = Xo − sXo · QXo , and quantize them into QRo . (3) Augmentation. We concatenate along dimension K as QXaug = [QX | QRo ] with combined scales sXaug = [sX | sRo ].” · Show in paper

“Offline Weight Quantization. Weights are aligned offline via: (1) Reordering. W is reordered to match X and quantized into QW . (2) Augmentation. Instead of computing residuals, we duplicate the quantized outlier weights QWo . We construct QWaug = [QW | QWo ], ensuring the GEMM effectively computes the correction term RoQ(Wo)⊤.” · Show in paper

Fused Quantization Kernel with Interleaved Channel Layout: A custom CUDA kernel fuses Channel Reordering, RMSNorm, Primary Quantization, and Residual Quantization into one operation, writing 16-channel primary blocks immediately followed by their corresponding 16-channel residual blocks in physical memory for coalesced access and compatibility with unmodified CUTLASS NVFP4 GEMM kernels.

Source excerpts

“To address this, we implement a Fused Quantization Kernel that integrates Channel Reordering, RMSNorm, Primary Quantization, and Residual Quantization into a single operation. We utilize coalesced memory access patterns during reordering to minimize latency, and output data in the exact Interleaved Channel Layout (detailed in Appendix D) for the subsequent GEMM.” · Show in paper

“Specifically, for the S outlier channels requiring compensation, we group them into blocks of 16 (matching the NVFP4 block size). Instead of separating all main quantizations and residual quantizations into distant memory regions, we interleave them locally: a 16-channel primary block is immediately followed by its corresponding 16-channel residual block in physical memory.” · Show in paper