1 Why This Matters
Before this paper was published in early 2020, training massive neural networks was largely guided by intuition, trial-and-error, and folklore. Machine learning practitioners frequently debated how to balance model size, training dataset size, and computational budgets. Should we train smaller models for longer on more data, or build massive models and train them briefly?
This paper transformed language modeling from an empirical art form into a predictable engineering science. It proved that cross-entropy loss scales smoothly and predictably as a power-law. By establishing these scaling relationships, the authors provided a roadmap for building the next generation of LLMs (including GPT-3 and GPT-4), showing exactly how much performance would improve with more compute and parameters.
2 The Big Idea
The core contribution of this work is that scale dominates performance. When not bottlenecked by other factors, the cross-entropy loss ($L$) of a Transformer language model scales as a power-law with the number of non-embedding parameters ($N$), the dataset size ($D$), and the amount of training compute ($C$).
Crucially, other architectural details—such as network width, depth, number of attention heads, or feed-forward ratio—have remarkably little effect on performance within a wide range, provided the total parameter budget is held constant. The paper famously summarizes this with a series of simple equations:
3 How It Works
To isolate the effects of $N$ and $D$, the authors trained a vast sweep of Transformer models varying from $10^3$ to $1.5 \times 10^9$ non-embedding parameters, and on datasets ranging from $22$ million to $23$ billion tokens.
They discovered that the joint relationship governing both parameter size and dataset size can be modeled by a unified equation that predicts the onset of overfitting:
$$L(N, D) = \left[ \left(\frac{N_c}{N}\right)^{\frac{\alpha_N}{\alpha_D}} + \frac{D_c}{D} \right]^{\alpha_D}$$
This equation shows that if we scale $N$ and $D$ together, performance improves smoothly. However, if we hold one fixed and scale the other, we hit a wall of diminishing returns.
Widget 1: Scaling Law Playground
Adjust the non-embedding parameters ($N$) and dataset size ($D$) to see how the predicted cross-entropy loss ($L$) behaves. Watch how holding one parameter fixed while increasing the other leads to overfitting bottlenecks.
4 Optimal Allocation of Compute
If you are given a fixed training compute budget ($C$), how should you allocate it between model size ($N$) and training steps ($S$)?
The paper answers this with a groundbreaking conclusion: when compute budget increases, we should scale model size aggressively, while scaling dataset size and training steps much more slowly.
Specifically, as compute $C$ increases, the optimal parameters grow as:
This means that for every 10x increase in compute, the model size should increase by roughly 5.4x, whereas the dataset size only needs to increase by 1.8x, and the number of optimization steps remains almost constant (1.07x).
Widget 2: Optimal Compute Allocator
Slide the total training compute budget ($C$) to see how a compute-optimal researcher would partition it. See how the model size ($N$) dominates the budget as scale increases.
5 Context Length & Architecture Efficiency
How do Transformers compare to older architectures like LSTMs? The paper demonstrates that while LSTMs perform comparably on short context lengths, they quickly hit a wall because they cannot leverage long-range dependencies.
Transformers, on the other hand, continue to improve their token-by-token prediction accuracy as the context window grows up to 1024 tokens and beyond.
Widget 3: Context Window Performance
Toggle the context length index to see how LSTM models plateau early, while Transformers continue to reduce loss across the entire token context.
6 Limitations & Open Questions
While Kaplan et al.'s scaling laws laid the foundation for modern AI scaling, subsequent research highlighted crucial limitations:
- The Chinchilla Correction: In 2022, Hoffmann et al. (DeepMind) published the "Chinchilla" scaling laws, showing that Kaplan et al. had underestimated the optimal amount of training data. Chinchilla proved that for optimal compute-efficiency, parameters and tokens should scale in equal proportion ($N \propto \sqrt{C}$ and $D \propto \sqrt{C}$).
- Inference Costs: The paper focuses strictly on *training* compute efficiency. However, in production environments, inference costs dominate. It is often much more cost-effective to train a smaller model far past its compute-optimal point to save massive amounts of compute during deployment.
- Qualitative Capabilities: Cross-entropy loss is a smooth curve, but downstream capabilities (such as reasoning, coding, and translation) often appear as emergent, non-linear jumps as models scale.
7 Glossary
Cross-Entropy Loss (nats)
Power-Law
PF-days (Petaflop-days)
Cite This Paper
@article{kaplan2020scaling,
title={Scaling Laws for Neural Language Models},
author={Kaplan, Jared and McCandlish, Sam and Henighan, Tom and Brown, Tom B and Chess, Benjamin and Child, Rewon and Gray, Scott and Radford, Alec and Wu, Jeffrey and Amodei, Dario},
journal={arXiv preprint arXiv:2001.08361},
year={2020}
}