Weak-to-Strong Generalization: Eliciting Strong Capabilities with Weak Supervision
Authors: Collin Burns*, Pavel Izmailov*, Jan Hendrik Kirchner*, Bowen Baker*, Leo Gao*, Leopold Aschenbrenner*, Yining Chen*, Adrien Ecoffet*, Manas Joglekar*, Jan Leike, Ilya Sutskever, Jeff Wu* (OpenAI, 2023)
"How can weak supervisors control models much smarter than them? We study an analogy to this problem: can we use weak models to supervise strong models, and how can we elicit the strong model's full capabilities?"
The Challenge
Superhuman models will perform tasks too complex for humans to reliably evaluate, breaking traditional RLHF.
The Discovery
Strong models naively finetuned on weak labels consistently outperform their supervisors, but a wide gap remains.
The Solution
An auxiliary confidence loss can recover up to 80% of the performance gap on NLP tasks by preventing error imitation.
Why This Matters
Today, we align advanced AI systems using RLHF. This works well because humans are currently smarter than models and can easily tell if a response is helpful, honest, or safe.
But what happens when AI systems become vastly more capable than any human expert? If a superhuman coding model writes a million lines of highly complex software, no human can reliably verify if it contains hidden backdoors or malicious code. Humans will only be able to provide weak supervision.
This creates a fundamental technical hurdle: How do we ensure a system much smarter than us remains aligned to our intent, even when we cannot fully comprehend its actions?
The Big Idea
Instead of waiting for superhuman models to arrive, the authors propose a clever, empirical analogy that can be studied today: use weak models to supervise strong models.
By training a highly capable student model (like GPT-4) on labels generated by a weak supervisor (like GPT-2), we can measure how well the strong model generalizes beyond its flawed training signals. If the strong model successfully generalizes, it suggests that alignment is not about "teaching" the model new capabilities, but rather "eliciting" what it already implicitly knows from pretraining.
How It Works
1. The Weak-to-Strong Pipeline
The setup consists of three main stages. Hover over each block in the diagram below to understand its role and how the metrics are calculated:
Weak Supervisor
Finetuned on ground truth labels. Generates weak predictions.
Strong Student
Finetuned on the weak labels. Goal: exceed the supervisor.
Strong Ceiling
Strong student trained directly on ground truth. The upper bound.
Interactive Pipeline Explorer
Hover over or click any of the boxes above to inspect how the weak-to-strong framework evaluates alignment generalization!
2. Performance Gap Recovered (PGR)
To quantify how much of the strong model's latent capability was successfully elicited, the authors define the Performance Gap Recovered (PGR) metric:
- A PGR of 1 (100%) means the student model completely recovered its maximum capability, ignoring the supervisor's errors.
- A PGR of 0 means the student performed exactly as poorly as its weak supervisor.
3. Overcoming Error Imitation: Auxiliary Confidence Loss
A major failure mode is that the strong model might simply learn to imitate the weak supervisor's systematic mistakes. To mitigate this, the authors introduce an auxiliary confidence loss that encourages the student to make confident predictions, even when they contradict the supervisor:
Where \(\alpha\) is a dynamic weight, \(f_w(x)\) is the weak supervisor's prediction, and \(\hat{f}_t(x)\) is the student's own hardened prediction.
Interactive Playground
Loss Function Mixer Simulation
Adjust the slider to see how the target label distribution shifts between the weak supervisor's prediction and the student's own confident prediction.
Training Overfitting & Early Stopping Simulator
Select a method and supervisor gap to see how overfitting to supervisor mistakes degrades performance over the training epoch.
Insight: Under naive training with a large gap, the student model initially learns the correct concept but quickly starts overfitting to the weak supervisor's systematic errors, causing test accuracy to plummet before the end of the epoch.
Headline Results
The authors evaluated their approaches across three diverse task domains: NLP Benchmarks, Chess Puzzles, and a proprietary ChatGPT Reward Modeling dataset.
~80% PGR
Using the auxiliary confidence loss, a strong student (GPT-4) supervised by a weak model (GPT-2 level) recovers nearly 80% of the performance gap.
Bootstrapping
For chess puzzles, direct weak-to-strong transfer is poor. However, bootstrapping through intermediate model sizes drastically improves generalization.
~15% PGR
Generalization is poorest on ChatGPT Reward Modeling. Naive methods only recover ~10%, indicating that standard RLHF might scale poorly to superhuman models.
Limitations & Disanalogies
While the empirical results are promising, the authors highlight critical disanalogies between their experimental setup and the actual challenge of aligning superhuman models:
Imitation Saliency
Superhuman models will likely have highly salient representations of human behavior and human errors. It might be much easier for a future superhuman model to perfectly imitate human mistakes than it is for GPT-4 to imitate GPT-2's errors.
Pretraining Leakage
Our current strong models have already seen human-level performance implicitly in their pretraining corpora. True superhuman capabilities might be completely latent or self-supervised, making them harder to elicit.
Glossary
Superalignment
PGR (Performance Gap Recovered)
Bootstrapping
Citation
@article{burns2023weak,
title={Weak-to-strong generalization: Eliciting strong capabilities with weak supervision},
author={Burns, Collin and Izmailov, Pavel and Kirchner, Jan Hendrik and Baker, Bowen and Gao, Leo and Aschenbrenner, Leopold and Chen, Yining and Ecoffet, Adrien and Joglekar, Manas and Leike, Jan and Sutskever, Ilya and Wu, Jeff},
journal={arXiv preprint arXiv:2312.09390},
year={2023}
}