By Pengxiang Zhao, Xing Li, Hui-Ling Zhen, Weizhe Lin, Xianzhi Yu, Zhenhua DongSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2611-16426.git
Cite
Version link
https://openversions.com/index/arxiv-2611-16426/commit/a2d598f39238299dee781e91f503cca069c2fe9f
Version identifier (commit hash)
a2d598f39238299dee781e91f503cca069c2fe9f

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Claims

Findings

Headline finding

Across Qwen3-8B and Qwen3-14B under W4A4 quantization with MXFP4, NVFP4, and HiF4, a SoScale variant achieves the best average downstream accuracy in every model-format group, preserving 96.44% to 97.17% of BF16 average accuracy under MXFP4, 98.99% to 99.20% under NVFP4, and 98.38% to 99.40% under HiF4, outperforming RTN, GPTQ, local ScaleSearch, and BlockRotation baselines.

Default range-preserving (maximum-absolute-value) scales in hierarchical FP4 quantization can reserve scarce FP4 levels for outlier regions that receive no assignments, reducing resolution for dense weight values; while local element-wise scale search reduces element-wise error, it does not consistently reduce model-level perplexity, whereas SoScale does.

Combining block Hadamard preprocessing with SoScale is format-dependent: it yields the best average score for both Qwen3-8B and Qwen3-14B under MXFP4 and for Qwen3-14B under NVFP4, whereas SoScale alone is best for Qwen3-8B under NVFP4 and Qwen3-14B under HiF4.

Under a direct-RTN activation quantization path without activation-side scale search, the best SoScale-w variant outperforms ScaleSearch-w in every model-format group on Qwen3-8B and Qwen3-14B, with gains ranging from 0.08 to 1.92 percentage points in average accuracy.

Adding GPTQ-style error recovery to a local MSE scale search yields mixed gains over ScaleSearch on Qwen3-8B (marginal under MXFP4, worse under NVFP4, and improved under HiF4), whereas full SoScale (coupling Hessian-aware scale-code selection with error recovery) achieves the best accuracy across all three formats.

Methods

Hessian-Aware Scale-Code Search: For each linear layer with calibration inputs X∈RT×n\boldsymbol{X} \in \mathbb{R}^{T \times n} and weights W∈Rm×n\boldsymbol{W} \in \mathbb{R}^{m \times n}, SoScale computes the empirical Hessian H=X⊤X\boldsymbol{H} = \boldsymbol{X}^\top \boldsymbol{X} and selects legal scale codes ci,b⋆=arg⁡min⁡c∈Ci,bei,b(c)⊤Hbei,b(c)\boldsymbol{c}^\star_{i,b} = \arg\min_{\boldsymbol{c} \in \mathbb{C}_{i,b}} \boldsymbol{e}_{i,b}(\boldsymbol{c})^\top \boldsymbol{H}_b \boldsymbol{e}_{i,b}(\boldsymbol{c}) for each row-block (i,b)(i, b), where Hb=HIb,Ib\boldsymbol{H}_b = \boldsymbol{H}_{\mathbb{I}_b, \mathbb{I}_b} and ei,b(c)=wi,b−w^i,b(c)\boldsymbol{e}_{i,b}(\boldsymbol{c}) = \boldsymbol{w}_{i,b} - \widehat{\boldsymbol{w}}_{i,b}(\boldsymbol{c}).

Source excerpts

“SoScale scores c by ℓi,b(c) = ei,b(c)⊤Hbei,b(c), (7) ... For each row-block pair (i, b), SoScale selects c⋆i,b = arg min c∈Ci,b ℓi,b(c), (8)” · Show in paper

Sequential Second-Order Error Recovery: SoScale computes an upper-triangular factor U\boldsymbol{U} of the damped inverse Hessian U⊤U=(H+λI)−1\boldsymbol{U}^\top \boldsymbol{U} = (\boldsymbol{H} + \lambda \boldsymbol{I})^{-1}, searches scale codes on the compensated working weights W~:,Q\widetilde{\boldsymbol{W}}_{:,\mathbb{Q}}, commits the quantized block W^:,Q=QF(W~:,Q;cQ⋆)\widehat{\boldsymbol{W}}_{:,\mathbb{Q}} = Q_{\mathcal{F}}(\widetilde{\boldsymbol{W}}_{:,\mathbb{Q}}; \boldsymbol{c}^\star_{\mathbb{Q}}), and propagates the block error EQ=W~:,Q−W^:,Q\boldsymbol{E}_{\mathbb{Q}} = \widetilde{\boldsymbol{W}}_{:,\mathbb{Q}} - \widehat{\boldsymbol{W}}_{:,\mathbb{Q}} to future blocks via W~:,F←W~:,F−EQUQ,Q−1UQ,F\widetilde{\boldsymbol{W}}_{:,\mathbb{F}} \leftarrow \widetilde{\boldsymbol{W}}_{:,\mathbb{F}} - \boldsymbol{E}_{\mathbb{Q}} \boldsymbol{U}_{\mathbb{Q},\mathbb{Q}}^{-1} \boldsymbol{U}_{\mathbb{Q},\mathbb{F}}.

Source excerpts

“Following GPTQ, we compute an upper-triangular factor U of the damped inverse Hessian, U⊤U = (H + λI)−1, (9) ... The block error is then propagated to the future blocks through the back-substitution: Wf:,F ← Wf:,F − EQU−1Q,QUQ,F. (13)” · Show in paper

Dynamic Per-Token Activation Quantization: Activations are quantized online per token using a lightweight format-specific dynamic scale rule ct,bact=DF(xt,b)c^{\text{act}}_{t,b} = D_{\mathcal{F}}(\boldsymbol{x}_{t,b}) selected via an element-wise MSE criterion, yielding x^t,b=QFact(xt,b;ct,bact)\widehat{\boldsymbol{x}}_{t,b} = Q^{\text{act}}_{\mathcal{F}}(\boldsymbol{x}_{t,b}; c^{\text{act}}_{t,b}).

Source excerpts

“Each activation block is quantized as xbt,b = QactF (xt,b; cactt,b), cactt,b = DF(xt,b), (14) where DF is the format-specific dynamic scale rule for activations. Because activation scales are computed online for every token, DF is kept lightweight and selects cactt,b for each token block using an element-wise MSE criterion.” · Show in paper

Calibration and Evaluation Setup: Calibration uses 128 sequences of 2,048 tokens from the first WikiText shard; quantized models are evaluated with the Language Model Evaluation Harness (batch size 64) on GSM8K (5-shot exact match) and HellaSwag, MMLU, ARC-Challenge, ARC-Easy, BoolQ, and PIQA (zero-shot accuracy).

Source excerpts

“For methods that require calibration, we sample 128 sequences of 2,048 tokens from the first WikiText shard (Merity et al. 2016), following a widely used PTQ setup (Frantar et al. 2023).” · Show in paper

“We evaluate quantized models using the Language Model Evaluation Harness (Gao et al. 2024) with batch size 64 on GSM8K (Cobbe et al. 2021), HellaSwag (Zellers et al. 2019), MMLU (Hendrycks et al. 2021), ARC-Challenge and ARC-Easy (Clark et al. 2018), BoolQ (Clark et al. 2019), and PIQA (Bisk et al. 2020).” · Show in paper