Headline finding
Full-Stack FP4, a modular NVFP4 framework with separate recipes for projections, AdamW states, Root/Muon computation, and attention, achieves a 2.286 loss (a 0.838% gap compared to BF16's 2.267) on 3B pretraining with 64B tokens, with average zero-shot perplexity of 26.665 (vs. 26.675 for BF16) and 0.10 percentage points lower average accuracy, while native four-block measurements on one RTX 5090 show 2.50–2.83× Root speedups over optimized BF16 and 37.9–42.5% lower AdamW peak memory.
All stated loss, perplexity, accuracy, speedup, and memory numbers match the reported experimental results in Sections 4.3, 4.4, Table 4, and Table 5, with the scope of native hardware measurements (four blocks on one RTX 5090) clearly disclosed.
“On 3B pretraining with 64B tokens, BF16 and Full-Stack FP4 reach losses of 2.267 and 2.286, a 0.838% gap. Their average zero-shot perplexities are 26.675 and 26.665, respectively, with Full-Stack FP4 averaging 0.10 percentage points lower in accuracy. Native four-block measurements on one RTX 5090 show 2.50–2.83× Root speedups over optimized BF16 and 37.9–42.5% lower AdamW peak memory.” · Show in paper