1. Why This Matters
Large Language Models (LLMs) are surprisingly good at multi-step reasoning, but they suffer from a fatal flaw: hallucinations. In logical domains like mathematics, a single incorrect step propagates errors down the line, completely derailing the final answer.
Traditionally, models are trained using Outcome-supervised Reward Models (ORMs). An ORM is only given feedback on the final answer: is it correct or incorrect? However, this creates a massive credit assignment problem. If a model writes 10 steps of math, gets step 3 wrong, but accidentally stumbles onto the correct final answer through a second error, the ORM rewards the entire chain. This teaches the model to hallucinate and use spurious reasoning.
To solve this, we need a way to verify reasoning step by step.
2. The Big Idea
The core contribution of this paper is a rigorous comparison between outcome-supervised reward models (ORMs) and Process-supervised Reward Models (PRMs).
Instead of labeling only the final output, human annotators label the correctness of every single step in a model's reasoning chain. This precise feedback allows the PRM to pinpoint exactly where a logical error occurred.
"Process supervision provides more precise feedback, since it specifies the exact location of any errors that occur. It also has several advantages relevant to AI alignment: it is easier for humans to interpret, and it more directly rewards models for following a human-endorsed chain-of-thought."
3. How It Works
The process relies on three key pillars: a step-formatted generator, step-by-step human feedback, and a mathematical formulation to score complete paths.
The Math Behind Step Scoring
A generator produces a solution as a sequence of newline-delimited steps: $S = (s_1, s_2, \dots, s_M)$. For each step $s_t$, the PRM outputs a probability that the step is correct: $P(s_t \text{ is correct} \mid s_{1 \dots t-1})$.
To score a complete solution, the paper defines the PRM score as the probability that every step is correct. This is calculated as the product of step-level probabilities:
Interactive Widget 1: Step-by-Step Scoring Simulator
Adjust the correctness probability of each step to see how the final PRM score is calculated compared to an ORM's blind final-answer check.
Solve for $x$: $3x + 7 = 16 - x$
Considers accuracy of every single step.
Only looks at final step ($x = 2.25$). Vulnerable to false positives.
4. Interactive Playground
To truly grasp how process-level feedback prevents models from cheating, explore the two scenarios below. Toggle between a Correct Solution and a Spurious Solution (False Positive) to see how the PRM catches sneaky logical leaps that fool outcome-based verifiers.
Interactive Widget 2: Solution Inspector
Compare a correct execution against a flawed one that reaches the same final answer.
Find all real solutions to $x^2 - 5x + 6 = 0$.
5. Results & Scaling
The authors evaluated their models on the challenging MATH dataset. The key metric is the Best-of-N Search performance: the generator samples $N$ candidate solutions, the reward model ranks them, and the top-ranked solution is evaluated.
As $N$ grows, the probability of generating at least one correct solution increases. However, the reward model must be accurate enough to select it.
Interactive Widget 3: Best-of-N Scaling Curve
Adjust $N$ (number of generated samples) to observe how PRM consistently widens its lead over ORM and Majority Voting.
Key Observation: At small values of $N$, the gap is narrow. However, as $N$ increases (scaling test-time compute), the PRM's advantage widens significantly because it is robust against selecting false-positive solutions that fool the ORM.
Active Learning: 2.6x More Data Efficient
Labeling step-by-step solutions is incredibly expensive. To optimize human annotator time, the authors used Active Learning. Instead of labeling random solutions, they purposefully surfaced "convincing wrong-answer" completions—solutions that the current PRM rated highly but reached an incorrect final answer.
This targeted feedback loop made the process 2.6x more data-efficient, achieving the same performance as uniform sampling with less than half the human labels.
6. Limitations & Open Questions
- Generalization Beyond Math: It remains an open question how well process supervision scales to softer, less structured domains (like creative writing or code synthesis) where distinct "steps" are harder to define.
- Human Labeling Cost: Even with active learning, human step-level annotation is highly demanding and expensive.
- Out-of-Distribution Generalization: While the PRM generalizes well to recent STEM exams (AP Physics, Calculus, Chemistry), extreme shifts in problem style can still degrade performance.
7. Glossary
Process Supervision
Providing feedback on each individual step of a multi-step reasoning process, rather than only evaluating the final output.
Outcome Supervision
Evaluating only the final result of a model's generation (e.g., checking if the final numerical answer of a math problem is correct).
Active Learning
A machine learning training strategy where the model dynamically selects which data points would be most informative to label, rather than training on a static, randomly sampled dataset.
Alignment Tax
The potential cost in performance or capability that arises from training an AI system to be safe and aligned with human values. The paper shows process supervision has a negative alignment tax (improving both safety and capability).
Cite this Paper
@misc{lightman2023lets,
title={Let's Verify Step by Step},
author={Hunter Lightman and Vineet Kosaraju and Yura Burda and Harri Edwards and Bowen Baker and Teddy Lee and Jan Leike and John Schulman and Ilya Sutskever and Karl Cobbe},
year={2023},
eprint={2305.20050},
archivePrefix={arXiv},
primaryClass={cs.LG}
}