Headline finding
FP4Quant achieves up to 1.56× end-to-end throughput improvement over BF16 on NVIDIA H100 GPUs with Qwen-7B, revealing a trade-off between throughput and model quality.
Table 2 directly supports the 1.56× speedup at G = 256 under TP4 and demonstrates the throughput-quality trade-off across group sizes.
“Experiments on NVIDIA H100 GPUs with Qwen-7B demonstrate up to 1.56× end-to-end throughput improvement over BF16, revealing a trade-off between throughput and model quality.” · Show in paper