By Weiming Hu, Zihan Zhang, Haoyan Zhang, Chen Zhang, Cong Guo, Yu Feng, Tianchi Hu, Guanglin Li, Guipeng Hu, Junsong Wang, Jingwen LengSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2601-19213.git
Cite
Version link
https://openversions.com/index/arxiv-2601-19213/commit/31f8b648d6517f410c7f3c4e6f1631c2195ac1d8
Version identifier (commit hash)
31f8b648d6517f410c7f3c4e6f1631c2195ac1d8

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Claims

Findings

Headline finding

M2XFP narrows the 4-bit LLM quantization accuracy gap by achieving on average a 70.63% reduction in accuracy loss compared to MXFP4 and a 37.30% reduction relative to NVFP4 on LLM benchmarks, while delivering up to 1.91x speedup and 1.75x energy savings over state-of-the-art accelerators.

Low-bit MX formats suffer from significant accuracy degradation primarily because their shared power-of-two (E8M0) scaling cannot precisely align with the block maximum, and simply preserving the maximum element of the block in FP16 precision drastically reduces MXFP4's perplexity to nearly match FP4 and NVFP4.

In metadata-augmented MX quantization, element-level extra mantissa (Elem-EM) consistently dominates under a fixed shared scale in the 4.5–4.75 equivalent bit width (EBW) range, whereas subgroup-level extra mantissa (Sg-EM-2bit) surpasses Elem-EM in the 4.5–4.75 EBW region once an adaptive shared scale is enabled.

M2XFP achieves the lowest Wikitext v2 perplexity across all evaluated LLMs (LLaMA2-7B, LLaMA3-8B, LLaMA3-70B, Mistral-7B, Falcon-7B) except OPT-6.7B (where BlockDialect is better by 0.03), and recovers average accuracy loss on DeepSeek-R1-Distill-Qwen-1.5B from 12.12% (MXFP4) to 4.59%.

The M2XFP top-1 decode units and quantization engine account for only 0.26% of area and 0.36% of power overhead across all core components, while the M2XFP PE tile introduces only +4.0% area overhead over MXFP4.

Methods

M2XFP hybrid metadata-augmented microscaling format using group size 32 and subgroup size 8, assigning 2-bit subgroup-level extra mantissa (Sg-EM-2bit) with adaptive shared scale to static weights and 2-bit element-level extra mantissa for the top-1 element (Elem-EM-top1) to dynamic activations.

Source excerpts

“M2XFP adopts a hybrid design that assigns: (1) Weights apply Sg-EM-2bit format, enabling fine-grained subgroup-scale refinement through offline adaptive optimization, thereby improving bit efficiency while maintaining fidelity. (2) Activations apply Elem-EM-top1 format, which captures outliers within each subgroup in real time with minimal routing overhead.” · Show in paper

“For M2XFP, we configure a shared E8M0 scaling factor with group size 32 and subgroup size 8.” · Show in paper

Online activation quantization and bias-clamp encoding (Algorithm 1) that computes an E8M0 shared scale, quantizes elements to FP4-E2M1, selects the top-1 element per subgroup (breaking ties by lowest index), quantizes the top-1 element to FP6-E2M3, adds a binary bias of 1 and clamps to preserve the high 4 bits of FP4, and packs the lower 2 bits as metadata.

Source excerpts

“To implement this encoding, we first add a bias of 1 to the FP6 binary value, then clamp the result to ensure the high bits remain consistent with the original FP4 value. The lower 2 bits serve as metadata representing the additional mantissa precision.(Step 6 & 7)” · Show in paper

Offline weight quantization with hierarchical MSE minimization over subgroup-level 2-bit extra mantissa k∈{0,1,2,3}k \in \{0, 1, 2, 3\} and group-level exponent bias b∈{−1,0,1}b \in \{-1, 0, 1\} to select scales (1+k4)⋅2E+b(1 + \frac{k}{4}) \cdot 2^{E+b}.

Source excerpts

“The optimization first finds the optimal mantissa refinement k for each subgroup given a bias b, then selects the best group-level exponent bias b.” · Show in paper

Systolic-array accelerator integration comprising a Top-1 Decode Unit (16-entry FP4-to-UINT lookup table and three-level comparator tree), an Augmented PE Tile (baseline FP4 MAC, auxiliary MAC for ΔX\Delta X, and shift-and-add subgroup scale refinement), and a two-stage streaming Quantization Engine.

Source excerpts

“M2XFP introduces three specialized units to support its hybrid quantization format:(1) Top-1 Decode Unit identifies the maximum element in each subgroup and forwards its metadata to the processing elements (PEs). (2) Augmented PE Tile executes FP4 × FP4 multiply–accumulate (MAC) operations while incorporating additional mantissa and subgroup-scale refinements. (3) Quantization Engine performs online Elem-EM quantization of activations following the procedure in Sec. 4.4.” · Show in paper

“As shown in Fig. 12, the quantization engine is a two-stage pipeline responsible for online encoding of activations.” · Show in paper