Why This Matters
Recent breakthroughs in AI reasoning (such as OpenAI's o1 and DeepSeek-R1) rely heavily on RLVR. Typically, training these models requires massive datasets with thousands of complex math or coding problems to incentivize step-by-step thinking.
This paper flips the data-centric paradigm on its head by asking a provocative question: How much data is actually necessary to ignite reasoning?
The answer is surprising: Just one. By training on a single, well-chosen mathematical problem, a base model can unlock latent reasoning capabilities, correct its formatting issues, and learn to generalize across entirely different categories of mathematics.
The Big Idea
"We show that reinforcement learning with verifiable reward using one training example (1-shot RLVR) is effective in incentivizing the mathematical reasoning capabilities of large language models."
The core insight is that base LLMs already possess powerful reasoning capabilities buried deep within their pre-trained weights. Instead of teaching the model *new* math, 1-shot RLVR acts as a catalyst, aligning the model to generate structured, step-by-step Chain-of-Thought (CoT) responses.
How It Works
1. Verifiable Reward & GRPO
The framework uses Group Relative Policy Optimization (GRPO), a resource-efficient RL algorithm. For each prompt, the model generates a group of responses. The rewards are computed strictly based on whether the final answer (parsed from a LaTeX \boxed{} block) matches the ground truth.
The loss function optimizes three components:
- Policy Gradient Loss: Reinforces outputs that yield correct final answers compared to the group average.
- KL Divergence: Prevents the model from drifting too far from the reference policy.
- Entropy Loss: Crucial for 1-shot learning; it forces the model to explore diverse paths instead of collapsing into a single output.
2. Selecting the "Golden" Example
Not all mathematical examples are created equal. To find the most effective single training example, the authors proposed ranking problems using their Historical Variance Score:
Where $s_{i,e}$ is the average accuracy of example $i$ at epoch $e$. High variance indicates that the problem is at the boundary of the model's capability—sometimes solvable, sometimes not—making it highly informative for reinforcement learning.
Interactive Example Selector
Select a problem to see its characteristics and how it performs as a 1-shot trainer.
"The pressure P exerted by wind on a sail varies jointly as the area A of the sail and the cube of the wind's velocity V..."
Interactive Playground: Post-Saturation Generalization
One of the paper's most fascinating discoveries is Post-Saturation Generalization. When training on exactly one example, the training accuracy hits 100% almost immediately (by step 100). However, if you keep training, the test accuracy on unseen benchmarks continues to rise for over 1,500 steps!
Use the slider below to step through the training process and observe how the model's behavior and performance evolve.
Results & Ablations
Is the model just "grokking" or is it actually optimizing via Policy Gradients? The authors proved that Policy Gradient loss is the primary driver of 1-shot success, rather than weight decay or regularization.
Toggle the loss components below to simulate the ablation study results on MATH500.
Configure Loss Function
Simulated Performance Output
Limitations & Open Questions
While 1-shot RLVR is incredibly powerful, the authors highlight several limitations and areas for future exploration:
- Compute Cost: Although the training dataset is reduced to one example, the RL training process still requires a substantial number of rollouts and gradient updates (millions of rollouts over 1,500+ steps) to achieve generalization.
- Sensitivity to Example Selection: Training fails if the selected example is too difficult ($\pi_{1208}$) or has incorrect labels ($\pi_{1207}$). Finding the optimal example automatically remains an open challenge.
- Domain Generalization: The study focuses heavily on mathematical reasoning. Whether 1-shot RLVR works as effectively for coding, instruction following, or creative writing remains to be verified.
Glossary
RLVR (Reinforcement Learning with Verifiable Reward)
A framework where models are trained using reinforcement learning, but instead of relying on human feedback or complex neural reward models, the reward is computed directly by verifying the final answer (e.g., executing code or string-matching math answers).
GRPO (Group Relative Policy Optimization)
A variant of PPO that optimizes policies by comparing a group of outputs generated from the same prompt, eliminating the need for a separate critic model and significantly saving GPU memory.
Post-Saturation Generalization
A phenomenon where test performance on unseen evaluation sets continues to improve even after training accuracy on the training set has saturated at 100%.
Citation
@inproceedings{wang2025oneshotrlvr,
title={Reinforcement Learning for Reasoning in Large Language Models with One Training Example},
author={Wang, Yiping and Yang, Qing and Zeng, Zhiyuan and Ren, Liliang and Liu, Liyuan and Peng, Baolin and Cheng, Hao and He, Xuehai and Wang, Kuan and Gao, Jianfeng and Chen, Weizhu and Wang, Shuohang and Du, Simon Shaolei and Shen, Yelong},
booktitle={39th Conference on Neural Information Processing Systems (NeurIPS 2025)},
year={2025}
}