Headline finding
QQQ, a Quality Quattuor-bit Quantization method with 4-bit weights and 8-bit activations, achieves performance on par with existing state-of-the-art LLM quantization methods while significantly accelerating inference, achieving speed boosts up to 2.24x, 2.10x, and 1.25x compared to FP16, W8A8, and W4A16, respectively.
The reported perplexity, zero-shot accuracy, and vLLM throughput benchmarks on LLaMA models support the claimed parity with SOTA quantization methods and the specific speedup figures.
“Our extensive experiments show that QQQ achieves performance on par with existing state-of-the-art LLM quantization methods while significantly accelerating inference, achieving speed boosts up to 2.24 ×, 2.10×, and 1.25× compared to FP16, W8A8, and W4A16, respectively.” · Show in paper