By Siyu Ding, Mingchuan Ma, Jiabo Tong, Xingrun Xing, Ziming Wang, Guoqi LiSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2607-04422.git
Cite
Version link
https://openversions.com/index/arxiv-2607-04422/commit/a722099a8c9252adc77b12e3ff7d534f7d82a9b6
Version identifier (commit hash)
a722099a8c9252adc77b12e3ff7d534f7d82a9b6

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Latest version

Checks complete

Archive from arXiv Source: https://arxiv.org/abs/2607.04422v2 OpenVersions: cdb8ec76-f1ed-43a2-a755-8c6d68ad635f

Posted

Claims

Findings

Headline finding

Full-Stack FP4, a modular NVFP4 framework with separate recipes for projections, AdamW states, Root/Muon computation, and attention, achieves a 2.286 loss (a 0.838% gap compared to BF16's 2.267) on 3B pretraining with 64B tokens, with average zero-shot perplexity of 26.665 (vs. 26.675 for BF16) and 0.10 percentage points lower average accuracy, while native four-block measurements on one RTX 5090 show 2.50–2.83× Root speedups over optimized BF16 and 37.9–42.5% lower AdamW peak memory.

LoRA-SVD protects a compact projection subspace in BF16 while retaining full-shape NVFP4 computation, reducing the linear-only loss gap from 1.40% to 0.61% on 1B/20B-token pretraining.

LoRA-SVD reduces projection-level relative MSE for outputs, input gradients, and weight gradients by approximately 10% compared with direct NVFP4 across the observed 1B training trajectory, and achieves 29.8–33.0% lower final-output MSE across d = 512–2048 on real text.

Periodic SVD realignment every 2,048 steps at d = 4096/6144 costs 160.00/226.96 ms per trigger, amortizing to 0.078/0.111 ms per step (0.026%/0.018% of the measured complete step) with 0.031/0.039 GiB (0.3%) added peak allocation.

An ordered square-root, BF16 tile-mean, and 16-element Hadamard pipeline enables stable NVFP4 AdamW second-moment storage in an integrated 1B pretraining run, reducing second-moment reconstruction error by 54.3% at a 500M observation point (and ~60% across the 1B trajectory) while using 5.0625 bits/element.

Shape-dependent Newton–Schulz coefficients combined with p99 outlier clipping stabilize direct NVFP4 Root iterations without an auxiliary mixed-precision or SVD path, reducing relative L2L_2 iteration error from approximately 10% to approximately 3% and achieving 3.14×/3.23× speedup over BF16 and 4.32×/4.07× speedup over GRASP in isolated Root measurements on RTX 5090.

Mixed-precision FP4 attention (Q-Attn), which assigns NVFP4 to QK⊤QK^\top and dSdS paths while retaining PVPV, P⊤dOP^\top dO, and dOV⊤dOV^\top in BF16 and reusing quantized Q/K/VQ/K/V views, achieves a +0.47% loss gap over BF16 on 1B/20B-token pretraining (with 4/6 projections) and is 1.27× faster than BF16 SDPA with 51.1%/52.9% less incremental allocation on RTX 5090.

Methods

LoRA-SVD parameterizes a training-time linear weight as W=Wres+βL2L1W = W_{\text{res}} + \beta L_2 L_1, keeping the compact principal subspace and its activation path in BF16 while computing the full-shape residual in NVFP4, using a Cholesky-QR subspace-constrained backward pass that reuses quantized activations and periodic randomized SVD realignment every 2,048 steps.

Source excerpts

“LoRA-SVD parameterizes a training-time weight as W = Wres + βL2L1. It preserves a compact principal subspace and its activation path in BF16 while computing the full-shape residual in NVFP4.” · Show in paper

“To maintain alignment with the principal subspace, we use a Cholesky-QR constraint. ... Quantized activations are reused when forming LoRA gradients to avoid an additional high-precision activation read and large GEMM ... We realign the subspace every 2,048 steps using randomized SVD [19] and reset Wres, L2, L1 using the top-r components.” · Show in paper

Quantized-State AdamW (Q-AdamW) applies Hadamard mixing before stochastic rounding and adaptive 4/6 quantization for the first moment, and an ordered three-stage preprocessing pipeline—(1) square root, (2) BF16 tile mean subtraction, and (3) 16-element Hadamard transform—before NVFP4 storage for the second moment, inverting the transform, mean separation, and square upon reconstruction.

Source excerpts

“For the first moment mt, Hadamard mixing smooths the distribution before SR and adaptive 4/6 quantization. For vt, we use an ordered three-stage preprocessing pipeline before NVFP4 storage: (1) square root shortens the tail; (2) a BF16 tile mean separates the positive common component and leaves an approximately zero-mean residual; and (3) a 16-element Hadamard transform spreads localized energy and makes the residual more symmetric. ... Reconstruction inverts the transform and mean separation and then squares the result.” · Show in paper

NVFP4 Root (Q-Root) combines Root's shape-dependent Newton–Schulz coefficients with p99 outlier clipping and executes the Newton–Schulz matrix products directly in NVFP4 using RTN quantization, random Hadamard transforms, and adaptive 4/6 scaling, while applying Q-AdamW to ineligible parameters.

Source excerpts

“We instead combine Root’s shape-dependent coefficients with p99 outlier clipping and execute the NS matrix products directly in NVFP4. ... The first-order momentum follows the first-moment storage recipe in §3.3. Q-Root is applied only to eligible matrix parameters. Embeddings, lm_head, normalization and bias parameters, LoRA factors, and narrow parameters use Q-AdamW. ... All quantizations use RTN and adaptive 4/6 scaling.” · Show in paper

Mixed-Precision FP4 Attention (Q-Attn) stores and reuses quantized QQ, KK, and VV views across forward and backward passes, executing QK⊤QK^\top and dSdS products in NVFP4 while retaining softmax-sensitive PVPV, P⊤dOP^\top dO, and dOV⊤dOV^\top products in BF16.

Source excerpts

“Table 2 assigns NVFP4 to the QK⊤ and dS paths and retains the softmax-sensitive P V , P⊤dO, and dOV ⊤ products in BF16. Quantized views are reused between forward and backward to avoid precision mismatch and repeated memory traffic.” · Show in paper