NeurIPS 2020

Language Models are Few-Shot Learners

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal... Alec Radford, Ilya Sutskever, Dario Amodei (OpenAI)

TL;DR

By scaling an autoregressive language model to 175 billion parameters (GPT-3), we can perform diverse language tasks using purely text-based interaction (in-context learning) without any task-specific fine-tuning or gradient updates.

175B Parameters Few-Shot Learning 300B Tokens No Fine-Tuning

Why This Matters

Historically, applying machine learning to Natural Language Processing (NLP) followed a strict paradigm: Pre-train on a large, generic text corpus, and then fine-tune on a specific, human-labeled dataset for the target task (e.g., translation, sentiment analysis).

While highly successful, this paradigm has severe bottlenecks:

Data Hunger: Collecting thousands of high-quality labeled examples for every single task is expensive and often impractical.
Out-of-Distribution Failure: Fine-tuned models tend to overfit to narrow training sets, leading to poor generalization in real-world scenarios.
Human Incompatibility: Humans don't need thousands of examples to learn a new task; we perform tasks dynamically from a brief natural language instruction or a couple of demonstrations.

The Big Idea

What if we treated unsupervised pre-training as an implicit form of meta-learning? During pre-training, a massive autoregressive model absorbs a wide array of skills, formatting conventions, and reasoning abilities.

Instead of updating the model's weights at inference time, we can communicate the task structure entirely through the input text prompt. This is called in-context learning.

How It Works

The Prompting Spectrum

The paper explicitly defines and contrasts three distinct ways to perform in-context learning, alongside the traditional fine-tuning baseline:

In-Context Prompting Simulator

See how different prompting paradigms change the input structure fed to GPT-3.

Widget 1

Select Task

Prompting Mode

Constructed Prompt (Input Context)

Expected Model Completion

Model Scaling & Architecture

GPT-3 uses the same basic transformer architecture as GPT-2, with modified initialization, pre-normalization, and alternating dense and locally banded sparse attention patterns. The crucial differentiator is scale. To study scaling behavior, OpenAI trained 8 different model sizes, spanning three orders of magnitude.

Model Scaling Explorer

Toggle between the 8 model sizes trained for the GPT-3 study.

Widget 2

Parameters ($N_{params}$)

175.0B

Layers ($n_{layers}$)

Dim ($d_{model}$)

12,288

Batch Size

3.2M tokens

96 Attention Blocks Stacked

Key Results & Benchmarks

GPT-3 was evaluated on dozens of standard benchmarks. In many tasks, the 175B model in few-shot mode matches or even beats fine-tuned baselines. Let's look at three representative benchmarks:

Benchmark Performance Comparison

Compare GPT-3's zero, one, and few-shot results to the SOTA fine-tuned baselines.

Widget 3

GPT-3 Zero-Shot 76.2%

GPT-3 One-Shot 72.5%

GPT-3 Few-Shot 86.4%

Prior State-of-the-Art (Fine-Tuned) 68.0%

LAMBADA tests long-range dependencies in text. GPT-3 Few-Shot achieves 86.4% accuracy, outperforming the fine-tuned SOTA of 68.0% by a massive margin.

Limitations & Open Questions

Despite GPT-3's breakthrough capabilities, the authors outline several core limitations:

Structural Bias: Being a unidirectional autoregressive model, GPT-3 lacks the bidirectional contextual representation of models like BERT. This hurts performance on comparison tasks (e.g., NLI, sentence similarity).
Data Contamination: With a training corpus sourced largely from the open web (Common Crawl), some benchmark test sets inevitably leaked into the training data.
Sample Inefficiency: GPT-3 requires vastly more text during pre-training than a human reads in an entire lifetime to reach similar competence levels.
Common Sense Limits: It still struggles on basic physical reasoning (e.g., "If I put cheese in the fridge, will it melt?").

Glossary

In-Context Learning

The process of specifying a task inside the context window of a pre-trained language model using natural language instructions or demonstrations, without performing weight updates or gradient steps.

Autoregressive Model

A type of model that predicts the next token in a sequence based purely on the preceding tokens. It generates text sequentially from left to right.

Perplexity

A measurement of how well a probability model predicts a sample. Lower perplexity indicates the model is more confident and accurate in its predictions.

Citation

@article{brown2020gpt3,
  title={Language Models are Few-Shot Learners},
  author={Brown, Tom B and Mann, Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared and Dhariwal, Prafulla and Neelakantan, Arvind and Shyam, Pranav and Sastry, Girish and Askell, Amanda and others},
  journal={arXiv preprint arXiv:2005.14165},
  year={2020}
}