By Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, Xia HuSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2402-02750.git
Cite
Version link
https://openversions.com/index/arxiv-2402-02750/commit/4f58daf67dbf27f50a2244902ac7c381bfde4d11
Version identifier (commit hash)
4f58daf67dbf27f50a2244902ac7c381bfde4d11

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Claims

Findings

Headline finding

KIVI, a tuning-free 2bit KV cache quantization algorithm with hardware-friendly implementation, enables Llama, Falcon, and Mistral models to maintain almost the same quality while using 2.6x less peak memory (including model weight), which enables up to 4x larger batch size and brings 2.35x ~ 3.47x throughput on real LLM inference workload.

Key cache should be quantized per-channel because a few fixed channels exhibit very large magnitudes, and per-channel quantization confines the quantization error to each individual channel without impacting other normal channels.

Value cache should be quantized per-token because, although it has no obvious outlier pattern, it is used to calculate the attention output (a weighted sum of value caches with sparse attention scores), so per-token quantization confines the error inside each individual token and leads to a much smaller relative attention output error.

Maintaining a full-precision KV cache sliding window (residual key and value cache XKrX_{K_r} and XVrX_{V_r}) for local relevant tokens is crucial for obtaining desirable performance on hard generation tasks such as mathematical reasoning (GSM8K).

For Llama and Mistral models, 2bit KIVI has only up to 2% accuracy drop on LM-Eval tasks (CoQA, TruthfulQA, GSM8K) and minimal impact on LongBench and Needle-in-a-Haystack retrieval, whereas 4bit KIVI is needed to maintain accuracy for Falcon-7B.

Changing the residual length from 128 to 32 does not result in a substantial performance gap, while group sizes 32 and 64 yield similar results and group size 128 significantly decreases performance on GSM8K.

On Llama-2-7B with ShareGPT-synthesized workloads on a single NVIDIA A100 GPU (80GB), 2bit KIVI enables up to 4x larger batch size and 2.35x ~ 3.47x throughput with similar peak memory usage compared to the 16bit baseline.

Methods

Asymmetric round-to-nearest integer quantization of KV cache: quantizing key cache per-channel and value cache per-token using Q(X)=⌊X−zXsX⌉Q(X) = \lfloor \frac{X - z_X}{s_X} \rceil and X′=Q(X)⋅sX+zXX' = Q(X) \cdot s_X + z_X, where zX=min⁡Xz_X = \min X and sX=(max⁡X−min⁡X)/(2B−1)s_X = (\max X - \min X)/(2^B - 1).

Source excerpts

“The B-bit integer quantization-dequantization process can be expressed as: Q(X) = ⌊(X - z_X)/s_X⌉, X' = Q(X) · s_X + z_X, where z_X = min X is the zero-point, s_X = (max X - min X)/(2^B - 1) is the scaling factor, and ⌊·⌉ is the rounding operation.” · Show in paper

Streaming group-wise quantization with a full-precision residual sliding window: splitting key cache into grouped part XKgX_{K_g} (quantized every GG tokens) and residual part XKrX_{K_r} (kept in full precision up to residual length RR), and similarly splitting value cache into XVgX_{V_g} and XVrX_{V_r} using a queue.

Source excerpts

“our key idea to solve this problem is to group key cache every G tokens and quantize them separately. Because the number of tokens in X_K can be arbitrary, we split X_K into two parts, namely, the grouped part XKgX_{K_g} = X_K[: l - r] and residual part XKrX_{K_r} = X_K[l - r :], where l is the number of tokens inside the current key cache X_K, r is the number of residual tokens, where l - r can be divisible by G.” · Show in paper

Hardware-friendly GPU implementation fusing dequantization with matrix multiplication (`Q_MatMul`) in CUDA and implementing the group-wise quantization kernel in Triton.

Source excerpts

“To minimize the overhead, we have fused the dequantization process with matrix multiplication, e.g., Q_MatMul in Figure 3, using CUDA. We also implement the group-wise quantization kernel in Triton.” · Show in paper

Evaluation on normal-context generation tasks (CoQA, TruthfulQA, GSM8K via LM-Eval), long-context tasks (LongBench and Needle-in-a-Haystack), and ShareGPT-synthesized inference workloads on a single NVIDIA A100 GPU (80GB).

Source excerpts

“Specifically, we adopt generation tasks from LM-Eval (Gao et al., 2021) for normal context length evaluation and LongBench (Bai et al., 2023) for long context evaluation, respectively.” · Show in paper

“To evaluate the wall-clock time efficiency of KIVI, following vLLM (Kwon et al., 2023), we synthesize workloads based on ShareGPT (sha, 2023), which contain input and output texts of real LLM services.” · Show in paper