Scaling Laws — FlashPapers Concept

In One Sentence

Scaling laws describe how a model's cross-entropy loss decreases predictably as a power-law function of compute, model parameter count, and dataset size, allowing researchers to mathematically project performance and optimize resource allocation before training massive models.

Intuition

Training state-of-the-art deep learning models is incredibly expensive. Before committing millions of dollars of compute to a training run, you want to know: How smart will this model actually be?

Scaling Laws provide the answer. They reveal that neural network performance (measured by cross-entropy loss) is not a chaotic, unpredictable variable. Instead, it follows a clean, mathematical power law. If you double your compute budget, your loss drops by a highly predictable fraction.

Crucially, these laws show that architectural hyper-parameters—such as network depth versus width—have remarkably little impact on final performance within a wide range. What matters almost entirely are three key dimensions:

$N$ (Parameters): The capacity of the model.
$D$ (Dataset Size / Tokens): The amount of training data shown to the model.
$C$ (Compute): The total floating-point operations (FLOPs) used during training.

The core debate in scaling laws is how to balance $N$ and $D$ given a fixed compute budget $C$. Early formulations suggested scaling parameters rapidly while keeping data relatively modest. Later, refined experiments proved that for optimal performance, parameters and training tokens should scale in equal proportion (1:1).

Formal Definition

The Compute Constraint

The total training compute $C$ (in FLOPs) is approximately bound to the model parameter count $N$ and the number of training tokens $D$ by the relation:

$$C \approx 6ND$$

*Note: A forward pass requires $\approx 2N$ FLOPs per token, and the backward pass requires $\approx 4N$ FLOPs per token, totaling $6N$ FLOPs per token.*

Kaplan et al. (2020) Power-Law Formulation

When performance is limited only by parameter count $N$ (with infinite data) or dataset size $D$ (with infinite parameters), the cross-entropy loss $L$ scales as:

$$L(N) \approx \left(\frac{N_c}{N}\right)^{\alpha_N}; \quad L(D) \approx \left(\frac{D_c}{D}\right)^{\alpha_D}$$

Under a joint bottleneck, the loss is modeled as:

$$L(N, D) = \left(\frac{N_c}{N}\right)^{\alpha_N} + \left(\frac{D_c}{D}\right)^{\alpha_D}$$

Kaplan et al. estimated $\alpha_N \approx 0.076$ and $\alpha_D \approx 0.095$. Solving this system under the $C \approx 6ND$ constraint suggested that for every increase in compute, one should scale parameters aggressively: $N \propto C^{0.73}$ and $D \propto C^{0.27}$.

Hoffmann et al. (2022) / Chinchilla Scaling

Hoffmann et al. identified that Kaplan's models were severely data-starved. They proposed a refined loss estimator:

$$L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta}$$

Where $E$ represents the irreducible loss of the data distribution. Using empirical fits across over 400 baseline runs, they estimated:

$$\alpha \approx 0.34, \quad \beta \approx 0.28, \quad A \approx 406.4, \quad B \approx 410.7, \quad E \approx 1.69$$

Minimizing $L(N, D)$ subject to $C = 6ND$ yields the optimal scaling exponents:

$$N \propto C^a, \quad D \propto C^b \quad \text{where} \quad a = \frac{\beta}{\alpha + \beta} \approx 0.45, \quad b = \frac{\alpha}{\alpha + \beta} \approx 0.55$$

Implication: $a \approx b \approx 0.5$. To scale compute optimally, model size ($N$) and data size ($D$) must be scaled in equal proportion.

When to use & when not to

When to use

Budget Planning: Estimating the exact GPU hours and costs needed to achieve a target validation loss.
Hyperparameter Tuning: Prioritizing dataset expansion over hyperparameter search, since scaling laws show architecture details matter less than scale.
Overtraining Decisions: If downstream inference costs are high, you may choose to deliberately "overtrain" a smaller model beyond the Chinchilla limit to make serving cheaper.

When not to use

Emergent Downstream Tasks: Do not rely purely on scaling laws to predict performance on complex, multi-step reasoning tasks, as these can show sudden, non-linear "emergence" rather than smooth power-law improvements.
Small-Scale Regimes: Under $10^7$ parameters, overheads, optimization quirks, and tokenization details dominate, breaking clean power laws.
Data-Constrained Domains: If high-quality data is physically capped, you cannot scale compute-optimally and must overparameterize or focus on synthetic data generation.

History & Evolution

January 2020

Kaplan et al. (OpenAI)

Published "Scaling Laws for Neural Language Models", establishing the empirical power-law relationships governing cross-entropy loss. They argued that larger models are highly sample-efficient, suggesting that optimal training involves scaling model size aggressively while stopping significantly before convergence.

March 2022

Hoffmann et al. (DeepMind)

Published "Training Compute-Optimal Large Language Models" (the Chinchilla paper). They proved that prior models (like GPT-3 and Gopher) were severely undertrained. They demonstrated that for compute-optimal training, model size and dataset size should scale in equal 1:1 proportion.

June 2022

Wei et al.

Published "Emergent Abilities of Large Language Models", highlighting that while pretraining loss scales predictably, specific downstream capabilities can show sudden, non-linear jumps (emergence) at certain scale thresholds.

Interactive Compute Allocator

Adjust the compute budget slider to compare the recommended parameter count ($N$) and token count ($D$) under the Kaplan (2020) vs. Chinchilla (2022) regimes.

Compute Budget ($C$): 1.00e+24 FLOPs

1e21 FLOPs (~37 A100-days) 1e24 FLOPs (~37,000 A100-days) 1e26 FLOPs (~3,700,000 A100-days)

Kaplan et al. (2020) Regime

Optimal Parameters ($N$) --

Optimal Dataset Size ($D$) --

Result: Parameter-heavy, data-starved model.

Chinchilla (2022) Regime

Optimal Parameters ($N$) --

Optimal Dataset Size ($D$) --

Result: Balanced, compute-optimal allocation.

Pseudocode

chinchilla_scaling.py

def get_compute_optimal_allocation(C_flops: float) -> dict:
    """
    Calculates the compute-optimal parameter count (N) and dataset size (D)
    for a given compute budget (C) in FLOPs, based on Hoffmann et al. (2022).
    """
    # Exponents from the paper (Table 3, Approach 3)
    a = 0.45
    b = 0.55
    
    # Calibrated scaling constants
    # At C = 1e24 FLOPs, optimal N ≈ 74B, D ≈ 2.25T tokens
    N_const = 7.4e10 / (1e24 ** a)
    D_const = 2.25e12 / (1e24 ** b)
    
    N = N_const * (C_flops ** a)
    D = D_const * (C_flops ** b)
    
    # Verify constraint: 6 * N * D ≈ C
    actual_compute = 6 * N * D
    
    return {
        "parameters_N": int(N),
        "tokens_D": int(D),
        "verified_compute": actual_compute
    }

# Example usage for a 1e24 FLOP training run
allocation = get_compute_optimal_allocation(1e24)
print(f"Optimal Parameters: {allocation['parameters_N'] / 1e9:.1f} Billion")
print(f"Optimal Tokens: {allocation['tokens_D'] / 1e9:.1f} Billion")

Common Variants & Related Concepts

Inference-Optimal Scaling

If a model will be queried billions of times post-training, it is often cheaper overall to overtrain a smaller model (e.g., LLaMA style) past the Chinchilla limit to save downstream inference compute.

Downstream Scaling Laws

Predicting specific capabilities (like coding accuracy or translation benchmarks) rather than raw cross-entropy loss. These are typically noisier and less linear.

Emergent Phenomenon

The observation that certain complex capabilities do not scale smoothly, but instead appear abruptly once a model reaches a critical scale threshold.

Representative Flash Papers

ryhchkpe82

Language Models are Few-Shot Learners (GPT-3)

Demonstrated the practical execution of massive-scale pretraining. GPT-3 scaled to 175 billion parameters, validating the early Kaplan scaling law hypotheses.

yudgctsnk7

Attention Is All You Need — Explaining the Transformer

The foundational architecture that unlocked scaling laws. The Transformer's parallelizable nature allowed models to scale across orders of magnitude without hitting training bottlenecks.