Headline finding
KIVI, a tuning-free 2bit KV cache quantization algorithm with hardware-friendly implementation, enables Llama, Falcon, and Mistral models to maintain almost the same quality while using 2.6x less peak memory (including model weight), which enables up to 4x larger batch size and brings 2.35x ~ 3.47x throughput on real LLM inference workload.
The headline claim overgeneralizes: Section 4.2.2 explicitly notes that Falcon-7B needs 4bit KIVI to maintain accuracy and suffers a large accuracy drop with 2bit KIVI, and the 2.6x memory reduction and throughput gains were only measured on Llama-2-7B.
“From this analysis, we developed a tuning-free 2bit KV cache quantization algorithm named KIVI. With hardware-friendly implementation, KIVI can enable Llama, Falcon, and Mistral models to maintain almost the same quality while using 2.6× less peak memory (including model weight). This reduction in memory usage enables up to 4× larger batch size, bringing 2.35× ∼ 3.47× throughput on real LLM inference workload.” · Show in paper