LoRA: Low-Rank Adaptation of Large Language Models
Edward J. Hu*, Yelong Shen*, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen
TL;DR
Instead of retraining all parameters of a giant model (full fine-tuning), LoRA freezes the pre-trained weights and injects trainable rank decomposition matrices into the Transformer layers. This reduces trainable parameters by up to 10,000x and GPU memory requirements by 3x with zero added inference latency.
Why This Matters
Fine-tuning large pre-trained language models (like GPT-3 or RoBERTa) has become the standard way to adapt them to specific downstream tasks. However, as models grow to hundreds of billions of parameters, full fine-tuning becomes prohibitively expensive.
For example, deploying independent instances of a fine-tuned GPT-3 175B model requires storing and serving 175 billion parameters (~350GB) for each task.
Prior parameter-efficient adaptation techniques tried to solve this but introduced critical trade-offs:
Introduce sequential bottleneck layers that increase model depth. This breaks hardware parallelism and introduces significant inference latency (up to 30% slowdown on short sequences).
Reserves part of the sequence length for continuous virtual prompt tokens, reducing the available sequence length to process actual downstream tasks.
The Big Idea
LoRA is inspired by the concept of intrinsic dimension, which suggests that the parameter updates ($\Delta W$) during model adaptation actually reside in a subspace with a much lower "intrinsic rank".
Instead of updating the full-rank weight matrix $W_0 \in \mathbb{R}^{d \times k}$ directly, LoRA factorizes the update matrix $\Delta W$ into two low-rank matrices $B \in \mathbb{R}^{d \times r}$ and $A \in \mathbb{R}^{r \times k}$, where the rank $r \ll \min(d, k)$.
During training, the pre-trained weight matrix $W_0$ is completely frozen, while $A$ and $B$ are optimized.
How It Works
Let's visualize the difference between standard adapters and LoRA. Standard adapters are placed sequentially, adding latency. LoRA runs in parallel to the original frozen weights.
Most importantly, when deploying, we can explicitly compute $W = W_0 + BA$ and store it directly. This means no additional inference latency.
Interactive Architecture Comparison
LoRA runs in parallel to the frozen self-attention projection weights. The output is scaled and summed with the frozen output.
Weight Initialization & Scaling
To ensure that LoRA behaves exactly like the original pre-trained model at the start of training, the authors use a clever initialization strategy:
- Matrix $A$ is initialized with a random Gaussian distribution: $A \sim \mathcal{N}(0, \sigma^2)$
- Matrix $B$ is initialized to zero.
This ensures that $\Delta W = B \cdot A = 0$ at the beginning of training, so the model starts with zero adaptation bias.
The update is scaled by a constant factor $\frac{\alpha}{r}$ where $\alpha$ is a constant hyperparameter. This scaling helps reduce the need to re-tune hyperparameters when varying the rank $r$.
Parameter & Memory Playground
Adjust the model dimensions and rank to see how LoRA dramatically compresses the trainable parameters compared to full fine-tuning.
Calculated Parameters (Per Layer)
Reduction in Trainable Parameters
Subspace Similarity & Intrinsic Rank
How do we know that a low rank $r$ is actually sufficient? The authors analyzed the subspace similarity between the learned weight updates $\Delta W$ at different ranks (e.g., $r=8$ vs $r=64$).
They found that the top singular vectors of the low-rank updates heavily overlap. In fact, the subspace spanned by the top singular vectors of $r=8$ shares a significant overlap with that of $r=64$.
Subspace Similarity Heatmap Explorer
Hover over the cells to explore the overlap metric $\phi(i, j)$ between the singular vectors of $\Delta W_{r=8}$ and $\Delta W_{r=64}$.
Vector $i$ ($r=8$): -
Vector $j$ ($r=64$): -
-
Headline Results
Despite having 10,000x fewer trainable parameters, LoRA matches or exceeds the performance of full fine-tuning on massive benchmarks like GLUE and GPT-3 175B adaptation.
GPT-3 175B Downstream Task Performance
Limitations & Open Questions
- Batching Multi-task Inputs: While you can merge $BA$ into $W_0$ to eliminate latency, doing so means you can only serve one adapted task per model instance. If you need to batch inputs for different tasks in a single forward pass, you cannot merge the weights and must run the paths dynamically, which loses some inference speedups.
- Optimal Parameter Selection: The paper primarily adapts the attention weights ($W_q, W_v$). Investigating the efficacy of adapting MLP layers, LayerNorms, or biases remains an active area of exploration.
- Theoretical Bounds: Why exactly is the adaptation subspace so low-rank? A full theoretical understanding of low-rank adaptation dynamics is still an open scientific question.
Glossary
Intrinsic Dimension
The minimum number of parameters needed to represent a manifold or solve a learning task effectively. Even if a model has billions of parameters, the actual "concept space" it navigates during fine-tuning can often be described by a much smaller subspace.
Rank Decomposition
Factorizing a high-rank matrix into a product of lower-rank matrices (e.g., decomposing a $d \times d$ matrix into $d \times r$ and $r \times d$ matrices). This massively reduces storage and computational complexity.
Inference Latency
The time it takes for a model to output a prediction after receiving an input. In production applications, keeping inference latency as close to zero as possible is critical.
Cite This Work
@inproceedings{
hu2022lora,
title={LoRA: Low-Rank Adaptation of Large Language Models},
author={Edward J. Hu and Yelong Shen and Phillip Wallis and Zeyuan Allen-Zhu and Yuanzhi Li and Shean Wang and Lu Wang and Weizhu Chen},
booktitle={International Conference on Learning Representations},
year={2022},
url={https://openreview.net/forum?id=a73ALZduZg}
}