By Man Liu, Xingchen Liu, Hairui Zhao, Guangming Tan, Dingwen TaoSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2611-04323.git
Cite
Version link
https://openversions.com/index/arxiv-2611-04323/commit/a4782f32cc42724d4678749981daa0dc92b73a4b
Version identifier (commit hash)
a4782f32cc42724d4678749981daa0dc92b73a4b

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Claims

Findings

Headline finding

FP4Quant achieves up to 1.56× end-to-end throughput improvement over BF16 on NVIDIA H100 GPUs with Qwen-7B, revealing a trade-off between throughput and model quality.

FP4Quant consistently outperforms Tahquant in preserving model quality under FP4 communication compression on Qwen-7B, achieving substantially lower validation loss (2.2306 vs. 3.4848) under TP with G = 64 and near-lossless performance (loss 2.1525 vs. BF16's 2.1505) under PP.

FP4Quant exhibits clear sensitivity to micro-group size G, where increasing G leads to gradual degradation in TP and a sharp breakdown at large G (e.g., G = 512) in both TP and PP.

With G = 64, FP4Quant consistently preserves downstream performance on CMMLU and C-Eval (valid) across model-parallel settings (TP, PP, and MP) on Qwen-7B, matching or slightly exceeding BF16 on C-Eval under PP and MP, while outperforming Tahquant.

FP4Quant enables efficient low-bit communication with substantial acceleration under TP (1.40× to 1.56× speedup over BF16) while incurring negligible runtime overhead under PP (up to 1.01× speedup over BF16).

Methods

Micro-grouped dual-scale FP4 quantization partitions each communication tensor into contiguous micro-groups of G∈{32,64,128,256,512}G \in \{32, 64, 128, 256, 512\} elements, computes the second moment σk2=1G∑ixk,i2\sigma_k^2 = \frac{1}{G}\sum_i x_{k,i}^2 and adaptive scale αk=clip(Rσk2+λ,10−3,103)\alpha_k = \text{clip}\left(\frac{R}{\sqrt{\sigma_k^2 + \lambda}}, 10^{-3}, 10^3\right) for pre-rotation normalization x~k,i=αkxk,i\tilde{x}_{k,i} = \alpha_k x_{k,i}, and after FWHT applies a group-wise max-absolute scale sk=clip(max⁡i∣hk,i∣R,10−12,106)s_k = \text{clip}\left(\frac{\max_i |h_{k,i}|}{R}, 10^{-12}, 10^6\right) to quantize values via qk,i=CvtToFP4(clip(hk,isk,−R,R))q_{k,i} = \text{CvtToFP4}\left(\text{clip}\left(\frac{h_{k,i}}{s_k}, -R, R\right)\right).

Source excerpts

“Each communication tensor is partitioned into contiguous micro-groups of G elements, where G ∈ 32, 64, 128, 256, 512. Each group is processed independently to capture localized statistics, enabling fine-grained and adaptive scaling while preserving high-efficiency GPU execution.” · Show in paper

“After the FWHT transformation in Section 3.2, the transformed values hk,i are quantized using a group-wise max-absolute scale: sk = clip(maxi |hk,i| / R, 10−12, 106), qk,i = CvtToFP4(clip(hk,i / sk, −R, R)).” · Show in paper

Hybrid two-level FWHT pipeline divides a length-GG orthonormal Walsh–Hadamard transform into intra-warp stages (d<32d < 32) executed in registers via `__shfl_xor_sync` without shared memory or synchronization, and cross-warp stages (d≥32d \ge 32) executed via shared memory with synchronization barriers, followed by normalization by 1/G1/\sqrt{G}.

Source excerpts

“For intra-warp stages with butterfly distances d < 32, paired elements are exchanged through register shuffle operations without shared-memory level: n ← __shfl_xor_sync(0xffffffff, v, d), v ← n − v (tid & d) != 0, v + n otherwise. ... For cross-warp stages with d ≥ 32, intermediate results are exchanged through shared memory with synchronization barriers. Only log2(G/32) such stages are required, while groups with G ≤ 32 complete the entire transform using register-level operations. The final output is normalized by 1/√G to obtain the orthonormal transform..” · Show in paper

Warp-cooperative FP4 packing exchanges adjacent 4-bit codes qk,iq_{k,i} in registers via a stride-1 butterfly shuffle `hi_i = __shfl_xor_sync(0xffffffff, qk,iq_{k,i}, 1) & 0x0F`, allowing even lanes (i mod 2=0i \bmod 2 = 0) to assemble one byte b=(hii≪4)∣qk,ib = (\text{hi}_i \ll 4) | q_{k,i} and perform a single coalesced store, while reversing the process during decompression.

Source excerpts

“Adjacent threads exchange their codes through a stride-1 butterfly shuffle: hii = __shfl_xor_sync(0xffffffff, qk,i, 1) & 0x0F. (4) For even lanes (i mod 2 = 0), the shuffled value provides the high nibble from the adjacent thread, while the local code remains as the low nibble. Each even lane then assembles one byte as b = (hii ≪ 4)| qk,i and performs a single coalesced store, whereas odd lanes remain inactive.” · Show in paper