QLoRA: Efficient Finetuning of Quantized LLMs
By Tim Dettmers*, Artidoro Pagnoni*, Ari Holtzman, Luke Zettlemoyer (University of Washington)
TL;DR
"QLoRA reduces the memory footprint of finetuning a 65B parameter model from over 780GB to under 48GB (enabling training on a single GPU) while preserving full 16-bit finetuning task performance."
Why This Matters
Finetuning large language models (LLMs) is crucial for aligning them to specific tasks, instructions, or safety guidelines. However, full 16-bit finetuning of a 65B parameter model requires over 780 GB of GPU memory. This makes state-of-the-art LLM training prohibitively expensive, keeping it locked behind the doors of massive corporate data centers.
While model quantization (discretizing weights to lower-bit representations) can drastically reduce memory for inference, it historically broke down during training due to the high-precision gradients required for backpropagation. QLoRA solves this paradigm by keeping the base model in 4-bit, while routing high-fidelity gradients through a sparse set of 16-bit trainable adapters.
The Big Idea
QLoRA achieves high-fidelity 4-bit finetuning without performance degradation by introducing three core innovations:
- 4-bit NormalFloat (NF4): An information-theoretically optimal quantization data type designed specifically for normally distributed neural network weights.
- Double Quantization (DQ): A method that quantizes the quantization constants themselves, saving an additional 0.37 bits per parameter.
- Paged Optimizers: Utilizing NVIDIA Unified Memory to seamlessly page optimizer states between GPU and CPU, protecting against sudden memory spikes during gradient checkpointing.
Finetuning Paradigm Comparison
Full Finetuning
• Base weights: 16-bit (Active)
• Optimizer states: 32-bit
• Memory for 65B: >780 GB
Standard LoRA
• Base weights: 16-bit (Frozen)
• Adapters: 16-bit (Active)
• Memory for 65B: >130 GB
QLoRA
• Base weights: 4-bit NF4 (Frozen)
• Adapters: 16-bit (Active)
• Memory for 65B: <48 GB
How It Works
1. 4-bit NormalFloat (NF4) Quantization
Neural network weights typically follow a zero-centered normal distribution. Standard quantization techniques (like Uniform Integer or Floating Point) divide the weight space into equal-sized bins, which leaves many bins empty and others highly congested.
NF4 is information-theoretically optimal. It constructs quantization bins such that each bin has an equal number of expected values. This is achieved by estimating the quantiles of a theoretical $N(0, 1)$ distribution.
Interactive Quantizer Simulator
Compare how NF4 vs. Uniform Int4 maps normally distributed weights.
2. Double Quantization (DQ)
To prevent outliers from destroying quantization precision, weights are divided into blocks (e.g., block size of 64). Each block requires its own 32-bit quantization scale constant.
While critical, these scales add an overhead of $32 / 64 = 0.5$ bits per parameter. Double Quantization treats these scale constants themselves as inputs to a second 8-bit FP quantization step with a block size of 256. This shrinks the scale overhead to:
This saves roughly 3 GB of VRAM for a 65B model, allowing it to squeeze comfortably into standard enterprise GPUs.
3. Paged Optimizers
During training of extremely large models, memory spikes (especially during gradient checkpointing of long sequence lengths) often trigger fatal Out-Of-Memory (OOM) errors.
QLoRA integrates Paged Optimizers, which utilize NVIDIA Unified Memory to perform automatic page-to-page transfers between the GPU and CPU. When the GPU runs out of memory, active optimizer states are evicted to CPU RAM and paged back as needed during the backward pass update.
VRAM Footprint & Architecture Explorer
Toggle model parameters and features to see VRAM requirements and how QLoRA compares to standard methods.
Results & Guanaco
Using QLoRA, the authors trained the Guanaco model family. Guanaco 65B is the best-performing open-source model in their evaluation, reaching 99.3% of the performance of ChatGPT on the Vicuna benchmark while requiring only 24 hours of finetuning on a single GPU.
A key takeaway from the paper's empirical evaluation is that data quality is far more important than dataset size. For example, the 9k sample OASST1 dataset outperformed the 450k sample FLAN v2 dataset on chatbot benchmarks.
Chatbot Performance Arena
Explore tournament Elo ratings and head-to-head match simulations evaluated by GPT-4 and humans.
Vicuna Benchmark Elo
Head-to-Head Simulator
Prompt: ...
Guanaco Response: ...
Limitations & Open Questions
While QLoRA is a massive step forward for democratizing LLM finetuning, the authors highlight several open questions and limitations:
- Scaling to 33B/65B: Although QLoRA recovers 16-bit performance on smaller models, the authors did not fully establish if it matches full 16-bit finetuning performance at the 33B and 65B scales due to immense computational costs of running the full 16-bit baselines at scale.
- Benchmark Reliability: Chatbot benchmarks (like Vicuna) are highly subjective and prone to evaluation biases when judged by other LLMs (like GPT-4), which tend to favor their own outputs.
- Extreme Quantization: Can we push the limits further to 3-bit or even 2-bit base models with adapters, and still recover full performance?
Glossary
Quantization ▼
The process of mapping continuous high-precision floating-point numbers (like FP32 or FP16) to a discrete set of low-precision values (like Int4 or NF4) to reduce memory and speed up computation.
Low-Rank Adapters (LoRA) ▼
A Parameter-Efficient Fine-Tuning (PEFT) method that freezes the base model weights and injects trainable rank decomposition matrices into the attention layers, drastically reducing trainable parameter counts.
Quantiles ▼
Cut points dividing the range of a probability distribution into continuous intervals with equal probabilities. Used in NF4 to ensure optimal bin utilization for normally distributed weights.
Citation
@article{dettmers2023qlora,
title={QLoRA: Efficient Finetuning of Quantized LLMs},
author={Dettmers, Tim and Pagnoni, Artidoro and Holtzman, Ari and Zettlemoyer, Luke},
journal={arXiv preprint arXiv:2305.14314},
year={2023}
}