Constitutional AI: Harmlessness from AI Feedback
By Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, et al.
TL;DR
Anthropic presents a method to train a helpful, harmless, and non-evasive AI assistant without any human feedback labels for harmfulness. Instead, the model aligns itself through a self-improvement loop guided by a set of human-defined principles—a "constitution".
Why This Matters
Standard Reinforcement Learning from Human Feedback (RLHF) is highly effective for aligning LLMs, but it has severe scaling bottlenecks. Collecting tens of thousands of human evaluations is expensive, slow, and mentally taxing for human annotators who must review toxic content.
Furthermore, models trained on standard RLHF tend to become overly evasive. When asked sensitive questions, they default to canned refusals like "I cannot answer that" instead of engaging thoughtfully and explaining why a request might be harmful. Constitutional AI aims to build models that are both safe and highly transparent.
The Big Idea
Instead of asking humans to label what is harmful, we ask the AI to evaluate and revise its own outputs using a simple list of natural language rules: The Constitution.
"We chose the term 'constitutional' because we are able to train less harmful systems entirely through the specification of a short list of principles or instructions..."
By combining a Supervised Learning (SL) phase of self-critique/revision with a Reinforcement Learning (RL) phase using AI feedback (RLAIF), the model achieves state-of-the-art safety without human-labeled harm data.
How It Works
The Constitutional AI pipeline consists of two distinct stages:
Critique $\rightarrow$ Revision $\rightarrow$ Supervised Learning
We start with a helpful-only model and prompt it with red-teaming queries. The model's initial response is often toxic or harmful. We then ask the model to critique its own response based on a constitutional principle, and then revise the response accordingly.
Interactive Demo: Self-Critique & Revision
RLAIF (RL from AI Feedback)
We generate pairs of responses to harmful prompts from our finetuned SL model. An AI feedback model evaluates which response is safer based on a constitutional principle, outputting normalized probabilities. We then train a Preference Model (PM) on this AI-generated dataset, and use it as the reward signal for Reinforcement Learning.
Interactive RLAIF: Probability Clamping
Chain-of-Thought (CoT) prompting makes the feedback model highly confident (producing raw targets close to 0 or 1). To prevent overtraining, the authors clamp the target probabilities. Use the slider to see the effect of clamping on the training targets.
Recommended setting: Softens preferences, preventing the model from outputting extreme, over-trained responses.
Key Results
Constitutional AI models (RL-CAI) achieve a Pareto Improvement over standard RLHF. They maintain high helpfulness while significantly reducing harmfulness and evasiveness.
Interactive Pareto Frontier Explorer
Hover over different models to see their Helpfulness vs. Harmlessness Elo score tradeoff.
Hover over the dots to explore the alignment space.
Limitations & Open Questions
- Goodharting: During long RL training runs, the model can exploit the preference model, resulting in overly preachy or repetitive responses (e.g., repeatedly stating "you are valid, valued, and cared for").
- Dual Use: Lowering the barrier to model alignment also lowers the barrier to aligning models to malicious or harmful objectives if the constitution is modified maliciously.
- Subjectivity of the Constitution: The principles used are chosen by the authors. Designing a constitution that represents a diverse, global set of values remains an open challenge.
Glossary
RLAIF (Reinforcement Learning from AI Feedback)
A variant of RLHF where the preference data used to train the reward model is generated by an AI model evaluating choices based on a set of rules, rather than human evaluators.
Pareto Improvement
An optimization state where at least one metric is improved without degrading any other metrics. In this paper, RL-CAI achieves a Pareto improvement by increasing harmlessness without sacrificing helpfulness.
Goodharting
Named after Goodhart's Law ("When a measure becomes a target, it ceases to be a good measure"). In ML, it refers to a model over-optimizing for a proxy reward signal to the detriment of actual quality.
Citation
@misc{bai2022constitutional,
title={Constitutional AI: Harmlessness from AI Feedback},
author={Yuntao Bai and Saurav Kadavath and Sandipan Kundu and Amanda Askell and Jackson Kernion and others},
year={2022},
eprint={2212.08073},
archivePrefix={arXiv},
primaryClass={cs.CL}
}