Emergent Abilities of Large Language Models
Authors: Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, William Fedus.
TL;DR
Scale isn't just about doing existing tasks better; it's about unlocking capabilities that were previously impossible. This paper defines, catalogs, and analyzes "emergent abilities"—qualitative leaps in performance (such as multi-step arithmetic, translation, and logical reasoning) that are entirely absent in smaller models but suddenly appear once a critical threshold of compute and parameters is crossed.
Why This Matters
In deep learning, we have grown accustomed to predictability. Through scaling laws, researchers can accurately forecast how cross-entropy loss will decrease as training compute spans orders of magnitude.
However, these smooth curves do not translate directly to the downstream tasks that humans actually care about. For many complex, multi-step tasks, model performance remains near-random for a long time, showing no signs of life. Then, suddenly, as the model scale crosses a critical threshold, performance leaps upward.
This behavior makes future AI capabilities highly unpredictable. If capabilities emerge abruptly, we cannot easily foresee what a model twice the size of today's state-of-the-art will be capable of doing—or what risks it might pose.
The Big Idea
"Emergence is when quantitative changes in a system result in qualitative changes in behavior."
Drawing inspiration from physicist Philip Anderson's famous 1972 essay "More Is Different", the authors define an ability as emergent if it is not present in smaller-scale models but is present in larger-scale models. When plotted on a scaling curve (with compute or parameters on the x-axis and performance on the y-axis), these abilities show a clear phase transition: performance is flat and near-random until a threshold is reached, after which it rises dramatically above random chance.
How It Works & How Emergence Manifests
The paper surveys emergent abilities across dozens of prior works, categorizing them into two primary paradigms: few-shot prompted tasks and augmented prompting strategies.
1. The Few-Shot Phase Transition
In few-shot prompting, a model is given a small number of exemplar input-output pairs as a preamble, followed by an unseen test query. For simple tasks (like sentiment analysis), performance scales smoothly. But for complex tasks like multi-digit arithmetic or symbolic reasoning, smaller models fail completely.
Use the interactive simulator below to explore how different tasks undergo a sudden phase transition as you scale up training compute (measured in FLOPs).
Interactive Scaling Curve Simulator
Adjust the scale slider to see the phase transition on different benchmarks.
2. The Metric Illusion: Masking Smooth Progress
Is emergence a fundamental property of scale, or is it an artifact of how we measure success? The paper reveals a vital nuance: while downstream metrics like Exact Match or Accuracy show sharp, emergent phase transitions, the underlying cross-entropy loss (which measures the model's actual confidence) improves smoothly and continuously.
Consider a multi-step task like math addition. If a model has to predict all digits correctly to get points, a small increase in individual token accuracy leads to a massive, non-linear leap in overall sequence accuracy.
Use the slider below to see how a smooth, continuous improvement in the probability of predicting a single token correctly results in a sudden, seemingly "emergent" leap in sequence-level accuracy.
The Multi-Step Probability Effect
See how sequence accuracy ($P^{\text{steps}}$) behaves as individual token probability increases.
Sequence Completion Probability Chain
3. The Emergent Task Catalog
The paper lists dozens of emergent abilities. Some are unlocked via Few-shot Prompting, while others require Augmented Prompting (like Chain-of-Thought reasoning or using a scratchpad).
Use the interactive catalog below to filter and inspect specific emergent abilities, their minimum scale thresholds, and the models in which they were first observed.
Headline Scales & Results
The scale at which an ability emerges is not an absolute law of nature; it depends on data quality, model architecture, and training objectives. However, across the models evaluated (GPT-3, LaMDA, Gopher, Chinchilla, and PaLM), several clear thresholds emerge:
Limitations, Risks, & Open Questions
While emergent abilities are exciting, they present major challenges for safety, alignment, and predictability:
Emergent Risks
Just as capabilities emerge suddenly, so do risks. Societal hazards like bias, toxicity, truthfulness issues, and even deceptive alignment behaviors may remain hidden in smaller models only to surface abruptly as scaling continues.
Beyond Scaling
Is scale the only path? The authors highlight that once an ability is discovered in a massive model, subsequent research often finds ways to unlock it in smaller models via better data quality, architectural tweaks, or specialized fine-tuning.
Glossary
Phase Transition
A sudden qualitative change in a system's behavior resulting from a continuous quantitative change in its parameters (e.g., water turning into steam, or an LLM suddenly solving arithmetic).
Few-shot Prompting
A technique where a language model is shown a few examples of a task in its context window before being asked to complete a new instance. No weights are updated during this process.
Chain-of-Thought (CoT)
An augmented prompting strategy where the model is prompted to output intermediate reasoning steps before producing the final answer, significantly boosting performance on logical and mathematical tasks.
Citation
@article{wei2022emergent,
title={Emergent Abilities of Large Language Models},
author={Wei, Jason and Tay, Yi and Bommasani, Rishi and Raffel, Colin and Zoph, Barret and Borgeaud, Sebastian and Yogatama, Dani and Bosma, Maarten and Zhou, Denny and Metzler, Donald and Chi, Ed H. and Hashimoto, Tatsunori and Vinyals, Oriol and Liang, Percy and Dean, Jeff and Fedus, William},
journal={Transactions on Machine Learning Research},
year={2022},
url={https://arxiv.org/abs/2206.07682}
}