Why This Matters
With the rise of reasoning-centric models like OpenAI's o1 and DeepSeek-R1, the community has widely embraced Reinforcement Learning with Verifiable Rewards (RLVR) as a magic wand. The prevailing belief was that RLVR acts like traditional RL in games (e.g., AlphaGo), allowing models to explore, self-correct, and autonomously discover entirely new reasoning strategies that exceed their starting capabilities.
However, this paper issues a crucial reality check. If RLVR is genuinely expanding a model's reasoning capabilities, then an RL-trained model should be able to solve complex tasks that the base model could never solve, no matter how many times we sampled from it. By testing this hypothesis systematically, the authors expose a fundamental limitation in how LLMs currently learn through reinforcement.
The Big Idea
RLVR does not teach models new tricks; it simply makes them faster at performing the tricks they already know.
By using the pass@k metric (which measures if a model can find a correct answer in $k$ attempts), the authors show that as we increase $k$, the base model catches up and eventually surpasses the RL-trained model. In other words, the base model already possessed the correct reasoning paths in its probability distribution, but they were buried. RL training merely "sharpens" the distribution around these paths, improving $pass@1$ efficiency at the cost of pruning away the creative, diverse paths needed to solve harder problems (reducing the overall reasoning boundary).
How It Works
The authors validate their claims by analyzing RLVR across multiple dimensions: pass@k curves, search tree coverage, entropy collapse, and distillation. Let's explore these mechanisms interactively.
1. The Pass@k Crossover Phenomenon
The formula for calculating an unbiased estimator of $pass@k$ given $n$ samples ($n \ge k$) where $c_i$ are correct is:
Use the interactive chart below to see how RL training steps affect the $pass@k$ curve. Notice how higher training steps boost performance at $k=1$ but cause a performance drop at $k=256$ compared to the base model.
2. Entropy Collapse & Search Trees
Why does RLVR reduce the reasoning boundary? Because of Entropy Collapse. As the model maximizes rewards, it focuses all its probability mass on the few paths that yielded positive rewards during training. This prunes the "search tree" of the model.
Click the buttons below to simulate sampling paths from a Base Model (highly diverse, exploring wide branches) versus an RLVR Model (narrowly focused on a single correct path, but unable to reach alternative solutions if the main path fails).
3. Distillation vs. RLVR
Interestingly, the authors find that Distillation (e.g., training a student model on long CoT traces generated by a stronger teacher like DeepSeek-R1) behaves completely differently from RLVR.
While RLVR is bounded by the base model's capacity, distillation actually introduces *new* reasoning patterns and expands the model's boundary.
RLVR forces the model to allocate all its probability to a narrow subset of the base model's existing capabilities. No new patterns are learned.
Key Results
The paper evaluates six popular RLVR algorithms (PPO, GRPO, Reinforce++, RLOO, ReMax, DAPO) on several benchmarks. The results demonstrate that while RLVR yields massive gains at $pass@1$, the performance gap at $pass@256$ is consistently negative compared to the base model.
| Model / Algo (Qwen2.5-7B) | Omni-MATH Pass@1 | Omni-MATH Pass@256 | MATH500 Pass@1 | MATH500 Pass@256 |
|---|---|---|---|---|
| Base Model (Step 0) | 9.9% | 67.2% | 34.5% | 96.2% |
| GRPO (Step 450) | 42.5% | 64.3% ↓ | 76.3% | 95.4% ↓ |
| PPO | 27.2% | 65.8% ↓ | 75.2% | 97.2% ↑ |
| RLOO | 28.6% | 66.4% ↓ | 75.0% | 97.4% ↑ |
| Reinforce++ | 28.2% | 67.7% ↑ | 75.4% | 96.8% ↑ |
Limitations & Open Questions
While the paper presents a solid case against current RLVR paradigms, several questions remain open:
- Proprietary Models: The study is restricted to open-source models (Qwen, LLaMA, Mistral). Frontier proprietary models like OpenAI's o1 might use more sophisticated multi-turn agent interactions or process-supervised reward models (PRMs) that could bypass these limitations.
- Scale of RL Training: Does scaling RLVR training to millions of steps eventually trigger an "aha moment" where the model breaks out of its prior? The authors' experiments were limited to hundreds of steps.
- High-Level Exploration: The authors suggest that to truly unlock RL, we need exploration mechanisms operating at a higher level of abstraction (e.g., program-level or conceptual-level search) rather than naive token-level sampling.
Glossary
RLVR (Reinforcement Learning with Verifiable Rewards)
A framework where LLMs are optimized using reinforcement learning based on easily computable, objective rewards (e.g., whether a math answer matches the ground truth or a code block passes unit tests), eliminating the need for human preference labeling.
Pass@k Metric
An evaluation metric where a model is allowed $k$ independent attempts to solve a problem. The problem is considered solved if at least one of the $k$ attempts is correct. This measures the model's capacity boundary rather than its average-case performance.
Entropy Collapse
A phenomenon during RL training where the model's output distribution becomes highly peaked around a few high-reward paths, drastically reducing the diversity of its generated answers.
Citation
@article{yue2025limitofRLVR,
title={Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?},
author={Yue, Yang and Chen, Zhiqi and Lu, Rui and Zhao, Andrew and Wang, Zhaokai and Yue, Yang and Song, Shiji and Huang, Gao},
journal={arXiv preprint arXiv:2504.13837},
year={2025}
}