Scaling Laws for Neural Language Models

1 Why This Matters

Before this paper was published in early 2020, training massive neural networks was largely guided by intuition, trial-and-error, and folklore. Machine learning practitioners frequently debated how to balance model size, training dataset size, and computational budgets. Should we train smaller models for longer on more data, or build massive models and train them briefly?

This paper transformed language modeling from an empirical art form into a predictable engineering science. It proved that cross-entropy loss scales smoothly and predictably as a power-law. By establishing these scaling relationships, the authors provided a roadmap for building the next generation of LLMs (including GPT-3 and GPT-4), showing exactly how much performance would improve with more compute and parameters.

2 The Big Idea

The core contribution of this work is that scale dominates performance. When not bottlenecked by other factors, the cross-entropy loss ($L$) of a Transformer language model scales as a power-law with the number of non-embedding parameters ($N$), the dataset size ($D$), and the amount of training compute ($C$).

Crucially, other architectural details—such as network width, depth, number of attention heads, or feed-forward ratio—have remarkably little effect on performance within a wide range, provided the total parameter budget is held constant. The paper famously summarizes this with a series of simple equations:

Parameter-Limited

L(N) ≈ (N_c / N)^α_N

α_N ≈ 0.076, N_c ≈ 8.8 × 10¹³

Data-Limited

L(D) ≈ (D_c / D)^α_D

α_D ≈ 0.095, D_c ≈ 5.4 × 10¹³

Compute-Limited

L(C) ≈ (C_c / C)^α_C

α_C ≈ 0.050, C_c ≈ 3.1 × 10⁸

3 How It Works

To isolate the effects of $N$ and $D$, the authors trained a vast sweep of Transformer models varying from $10^3$ to $1.5 \times 10^9$ non-embedding parameters, and on datasets ranging from $22$ million to $23$ billion tokens.

They discovered that the joint relationship governing both parameter size and dataset size can be modeled by a unified equation that predicts the onset of overfitting:

$$L(N, D) = \left[ \left(\frac{N_c}{N}\right)^{\frac{\alpha_N}{\alpha_D}} + \frac{D_c}{D} \right]^{\alpha_D}$$

This equation shows that if we scale $N$ and $D$ together, performance improves smoothly. However, if we hold one fixed and scale the other, we hit a wall of diminishing returns.

Widget 1: Scaling Law Playground

Adjust the non-embedding parameters ($N$) and dataset size ($D$) to see how the predicted cross-entropy loss ($L$) behaves. Watch how holding one parameter fixed while increasing the other leads to overfitting bottlenecks.

Parameters ($N$) 1.0M

1k 1M 1B 100B

Dataset Size ($D$ in tokens) 10.0M

100k 10M 1B 100B

Predicted Cross-Entropy Loss

3.45 nats

Loss Landscape (Cross-Section)

4 Optimal Allocation of Compute

If you are given a fixed training compute budget ($C$), how should you allocate it between model size ($N$) and training steps ($S$)?

The paper answers this with a groundbreaking conclusion: when compute budget increases, we should scale model size aggressively, while scaling dataset size and training steps much more slowly.

Specifically, as compute $C$ increases, the optimal parameters grow as:

Model Size

N ∝ C^0.73

Dataset Size

D ∝ C^0.27

Training Steps

S ∝ C^0.03

This means that for every 10x increase in compute, the model size should increase by roughly 5.4x, whereas the dataset size only needs to increase by 1.8x, and the number of optimization steps remains almost constant (1.07x).

Widget 2: Optimal Compute Allocator

Slide the total training compute budget ($C$) to see how a compute-optimal researcher would partition it. See how the model size ($N$) dominates the budget as scale increases.

Compute Budget ($C$ in PF-days) 1.0 PF-day

10^-5 (Tiny) 10^-2 10⁰ (Base) 10² 10⁵ (Massive)

Optimal Model Size (N)

1.3B params

Optimal Dataset (D)

20.0B tokens

Optimal Batch Size (B)

2.0M tokens

Optimal Steps (S)

5,400 steps

Visualizing Resource Allocation Growth

Model Parameters (N) Dataset Size (D) Training Steps (S)

5 Context Length & Architecture Efficiency

How do Transformers compare to older architectures like LSTMs? The paper demonstrates that while LSTMs perform comparably on short context lengths, they quickly hit a wall because they cannot leverage long-range dependencies.

Transformers, on the other hand, continue to improve their token-by-token prediction accuracy as the context window grows up to 1024 tokens and beyond.

Widget 3: Context Window Performance

Toggle the context length index to see how LSTM models plateau early, while Transformers continue to reduce loss across the entire token context.

Token Position in Context

1 (Start) 512 1024 (End)

Transformer Loss: 3.10 nats

LSTM Loss: 3.80 nats

6 Limitations & Open Questions

While Kaplan et al.'s scaling laws laid the foundation for modern AI scaling, subsequent research highlighted crucial limitations:

The Chinchilla Correction: In 2022, Hoffmann et al. (DeepMind) published the "Chinchilla" scaling laws, showing that Kaplan et al. had underestimated the optimal amount of training data. Chinchilla proved that for optimal compute-efficiency, parameters and tokens should scale in equal proportion ($N \propto \sqrt{C}$ and $D \propto \sqrt{C}$).
Inference Costs: The paper focuses strictly on *training* compute efficiency. However, in production environments, inference costs dominate. It is often much more cost-effective to train a smaller model far past its compute-optimal point to save massive amounts of compute during deployment.
Qualitative Capabilities: Cross-entropy loss is a smooth curve, but downstream capabilities (such as reasoning, coding, and translation) often appear as emergent, non-linear jumps as models scale.

7 Glossary

Cross-Entropy Loss (nats)

A metric used to evaluate how well a language model predicts the next token. Measured in "nats" when using natural logarithms (base $e$). Lower values represent better performance.

Power-Law

A mathematical relationship where one quantity varies as a power of another (e.g., $y = a x^k$). On a log-log plot, power-law relationships show up as perfectly straight lines.

PF-days (Petaflop-days)

A unit of compute representing $10^{15}$ floating-point operations per second performed continuously for 24 hours (equivalent to approximately $8.64 \times 10^{19}$ total FLOPs).

Cite This Paper

@article{kaplan2020scaling,
  title={Scaling Laws for Neural Language Models},
  author={Kaplan, Jared and McCandlish, Sam and Henighan, Tom and Brown, Tom B and Chess, Benjamin and Child, Rewon and Gray, Scott and Radford, Alec and Wu, Jeffrey and Amodei, Dario},
  journal={arXiv preprint arXiv:2001.08361},
  year={2020}
}

Share this Flash Paper