By Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke ZettlemoyerSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2305-14314.git
Cite
Version link
https://openversions.com/index/arxiv-2305-14314/commit/ed7f3e0a818ae4c554fe55e263dbdd403db7a03c
Version identifier (commit hash)
ed7f3e0a818ae4c554fe55e263dbdd403db7a03c

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Claims

Findings

Headline finding

QLoRA reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance.

4-bit NormalFloat (NF4) is an information-theoretically optimal data type for normally distributed weights that yields better empirical performance than 4-bit Integers (Int4) and 4-bit Floats (FP4).

Double Quantization reduces the average memory footprint of quantization constants from 0.5 bits to 0.127 bits per parameter (a reduction of 0.373 bits per parameter, or approximately 3 GB for a 65B model) without degrading performance.

Applying LoRA to all linear transformer block layers is required to match 16-bit full finetuning performance, whereas standard practice of applying LoRA only to query and value attention projection matrices does not match 16-bit performance.

Guanaco 65B, trained with QLoRA on OASST1, is the best-performing open-source chatbot model, reaching 99.3% of ChatGPT's performance on the Vicuna benchmark while requiring only 24 hours of finetuning on a single GPU.

Data quality and suitability are far more important than dataset size for instruction finetuning and chatbot performance, and strong MMLU performance does not imply strong chatbot performance and vice versa.

GPT-4 evaluations are a cheap and somewhat reliable alternative to human evaluation for chatbot performance at the system level, though they exhibit noticeable biases (such as order effects and self-preference) and weaker sample-level agreement.

Methods

4-bit NormalFloat (NF4) quantization estimates the 2k+12^k + 1 quantiles of a theoretical N(0,1)N(0, 1) distribution, normalizes them to [−1,1][-1, 1], and constructs an asymmetric data type with an exact discrete zeropoint of 0.

Source excerpts

“The information theoretically optimal data type for zero-mean normal distributions with arbitrary standard deviations σ\sigma in the range [−1,1][-1, 1] is computed as follows: (1) estimate the 2k+12^k + 1 quantiles of a theoretical N(0,1)N(0, 1) distribution to obtain a k-bit quantile quantization data type for normal distributions, (2) take this data type and normalize its values into the [−1,1][-1, 1] range, (3) quantize an input weight tensor by normalizing it into the [−1,1][-1, 1] range through absolute maximum rescaling.” · Show in paper

“To ensure a discrete zeropoint of 0 and to use all 2k2^k bits for a k-bit datatype, we create an asymmetric data type by estimating the quantiles qiq_i of two ranges qiq_i: 2k−12^{k-1} for the negative part and 2k−1+12^{k-1} + 1 for the positive part and then we unify these sets of qiq_i and remove one of the two zeros that occurs in both sets.” · Show in paper

Double Quantization (DQ) quantizes the 32-bit quantization constants c2FP32c_2^{\text{FP32}} of the first quantization (blocksize 64) into 8-bit Floats c2FP8c_2^{\text{FP8}} with a blocksize of 256 and second-level 32-bit constants c1FP32c_1^{\text{FP32}} after mean-centering.

Source excerpts

“More specifically, Double Quantization treats quantization constants c2FP32c_2^{\text{FP32}} of the first quantization as inputs to a second quantization. This second step yields the quantized quantization constants c2FP8c_2^{\text{FP8}} and the second level of quantization constants c1FP32c_1^{\text{FP32}}. We use 8-bit Floats with a blocksize of 256 for the second quantization as no performance degradation is observed for 8-bit quantization, in line with results from Dettmers and Zettlemoyer [13].” · Show in paper

Paged Optimizers use NVIDIA unified memory to allocate paged memory for optimizer states, automatically evicting them to CPU RAM when the GPU runs out of memory and paging them back when needed in the optimizer update step.

Source excerpts

“We use this feature to allocate paged memory for the optimizer states which are then automatically evicted to CPU RAM when the GPU runs out-of-memory and paged back into GPU memory when the memory is needed in the optimizer update step.” · Show in paper

QLoRA stores pretrained weights in 4-bit NormalFloat (NF4), dequantizes them to 16-bit BrainFloat (BFloat16) during the forward and backward passes, and computes weight gradients only for the 16-bit LoRA adapter parameters.

Source excerpts

“To summarize, QLORA has one storage data type (usually 4-bit NormalFloat) and a computation data type (16-bit BrainFloat). We dequantize the storage data type to the computation data type to perform the forward and backward pass, but we only compute weight gradients for the LoRA parameters which use 16-bit BrainFloat.” · Show in paper

Chatbot performance is evaluated via tournament-style pairwise comparisons judged by both human annotators and GPT-4, aggregated into Elo ratings (starting at 1,000, K=32K = 32) over 10,000 random orderings.

Source excerpts

“We randomly sample from the set of labeled comparisons to compute Elo [16, 17]. ... We start with a score of 1,000 and use K = 32. Similar to Chiang et al. [10], we repeat this procedure 10,000 times with different random seeds to control for ordering effects, e.g., the effect of which model pairs compete with each other first.” · Show in paper