Headline finding
QLoRA reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance.
While QLoRA's memory reduction to <48GB for 65B models is supported, preserving full 16-bit finetuning performance is only verified up to 3B models; at 7B-65B scales, QLoRA is only compared against 16-bit LoRA, not full 16-bit finetuning, as acknowledged in Section 8.
“We present QLORA, an efficient finetuning approach that reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance.” · Show in paper