Training Compute-Optimal Large Language Models
Jordan Hoffmann*, Sebastian Borgeaud*, Arthur Mensch*, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, and colleagues.
TL;DR
Large language models have been significantly undertrained. By training over 400 models across various compute budgets, the authors show that for compute-optimal training, model size and training tokens should scale in equal proportions (1:1). This discovery birthed Chinchilla (70B), which outperforms much larger models like GPT-3 (175B) and Gopher (280B) while using the exact same compute budget.
Why This Matters
In the early 2020s, the race to build larger AI models was driven by the belief that parameter count was the primary lever for performance. This was heavily influenced by Kaplan et al. (2020), which suggested that if you increase your compute budget by 10×, you should scale your model size by 5.5× but only increase training tokens by 1.8×.
This led to massive, "undertrained" giants like GPT-3 (175B parameters trained on 300B tokens) and Gopher (280B parameters trained on 300B tokens). While these models were powerful, they were highly inefficient to run.
The Hidden Cost of Over-parameterization
A model's parameter count directly dictates its memory footprint and inference speed. Running inference on a 280B model requires multiple high-end GPUs. If we can achieve the same (or better) performance with a 70B model trained on more data, we dramatically reduce the cost of running that model in production forever.
The Big Idea
"For every doubling of model size, the number of training tokens should also be doubled."
By systematically training over 400 models ranging from 70 million to 16 billion parameters across diverse token horizons, DeepMind's researchers found that parameters ($N$) and data tokens ($D$) should scale at a 1:1 ratio.
Interactive Explorers
1. Compute Allocation Simulator
Adjust the training compute budget (in FLOPs) to see how Kaplan's scaling law compares to the Chinchilla optimal law.
Kaplan Scaling Law
Parameter-heavyFormula: $N \propto C^{0.73}$, $D \propto C^{0.27}$
Chinchilla Scaling Law
Compute-OptimalFormula: $N \propto C^{0.50}$, $D \propto C^{0.50}$
2. IsoFLOP Loss Valley Explorer
For a fixed budget of $10^{21}$ FLOPs, slide the parameter count to see how the trade-off creates a "valley" of optimal loss.
Parametric Loss Fit:
L(N,D) = E + A/Nα + B/Dβ
Where E=1.69, α=0.34, β=0.28
How It Works
The authors used three distinct empirical approaches to estimate the relationship between compute, model size, and training data.
Approach 1: Varying training tokens for fixed model sizes
The researchers trained a family of models (from 70M to 10B parameters) and varied the number of training tokens for each. For any given compute budget, they extracted the model size that achieved the absolute lowest loss. This yielded the power-law relationship: $$N_{\text{opt}} \propto C^a, \quad D_{\text{opt}} \propto C^b$$ where $a = 0.50$ and $b = 0.50$.
Approach 2: IsoFLOP profiles
The team created specific "IsoFLOP" profiles by fixing the total training compute (e.g., $6 \times 10^{18}$ FLOPs) and varying the model size. By plotting the final loss against the parameter count, they observed clear parabolic "valleys." The minimum of each parabola represents the most compute-efficient model size for that budget. This approach yielded $a = 0.49$ and $b = 0.51$.
Approach 3: Parametric loss fitting
Finally, they modeled the final loss of all 400+ runs as a parametric function of parameters ($N$) and tokens ($D$): $$\hat{L}(N, D) \triangleq E + \frac{A}{N^\alpha} + \frac{B}{D^\beta}$$ By fitting this function using L-BFGS, they estimated: $$\alpha = 0.34, \quad \beta = 0.28, \quad A = 406.4, \quad B = 410.7, \quad E = 1.69$$ Solving for the optimal allocation yields $a = \frac{\beta}{\alpha+\beta} \approx 0.45$ and $b = \frac{\alpha}{\alpha+\beta} \approx 0.55$.
The Learning Rate Schedule Secret
A critical flaw in Kaplan's original study was using a fixed learning rate schedule length. Because cosine learning rate schedules perform best when matched to the exact number of training steps, Kaplan's intermediate loss estimates were artificially inflated, leading them to underestimate the value of training models longer on more data.
Results
To prove their scaling laws, DeepMind trained Chinchilla (70B) using the same compute budget as their previous flagship model, Gopher (280B), but with 4× fewer parameters and 4.6× more data (1.4 Trillion tokens).
Model Performance Comparison
Compare Chinchilla against other landmark large language models.
Limitations & Open Questions
- Only Two Large Scale Runs: Due to extreme training costs, the authors only compared Gopher and Chinchilla directly at the largest scale. There are no intermediate scaling validation runs at 100B+ parameters.
- The Single-Epoch Limit: All scaling runs were trained on less than one epoch of data. Whether these scaling laws hold in multi-epoch regimes remains an open research question.
- Concavity at Extreme Compute: The authors observed slight concavity in the FLOP-loss frontier at very high compute budgets, suggesting that even smaller models might be optimal at extreme scales.
Glossary
FLOPs
Floating Point Operations. A measure of computational work. For a standard transformer, training compute is estimated as $C \approx 6ND$ FLOPs, where $N$ is parameters and $D$ is tokens.
IsoFLOP Profile
An experimental setup where the total computational budget (FLOPs) is held constant, and researchers vary the ratio of model parameters to training tokens to find the optimal balance that minimizes loss.
Cosine LR Schedule
A learning rate decay schedule following a cosine curve. It is highly effective in LLM training but must be properly calibrated to the total length of training steps to yield optimal loss.
Cite This Paper
@article{hoffmann2022training,
title={Training Compute-Optimal Large Language Models},
author={Hoffmann, Jordan and Borgeaud, Sebastian and Mensch, Arthur and Buchatskaya, Elena and Cai, Trevor and Rutherford, Eliza and de Las Casas, Diego and Hendricks, Lisa Anne and Welbl, Johannes and Clark, Aidan and others},
journal={arXiv preprint arXiv:2203.15556},
year={2022}
}