Why This Matters
When DeepSeek-R1-Zero took the AI world by storm, it suggested a simple, almost magical paradigm: take a raw, pretrained base model, apply reinforcement learning (RL) directly without any supervised fine-tuning (SFT), and watch complex reasoning behaviors—like self-reflection, backtracking, and the famous "Aha moment"—spontaneously emerge.
However, this narrative leaves deep, unanswered questions. Does RL truly inject reasoning capabilities from scratch, or does it merely unlock behaviors already baked into the pretraining data? Furthermore, does Group Relative Policy Optimization (GRPO), the standard RL algorithm used for R1, optimize cleanly, or does it introduce hidden biases that force models to overthink and write unnecessarily long responses?
The Big Idea
"We critically examine R1-Zero-like training by analyzing its two core components: base models and RL. We show that DeepSeek-V3-Base already exhibits self-reflection prior to RL, and identify a structural length bias in GRPO that artificially rewards bloated, incorrect answers. By removing these mathematical biases, our proposed Dr. GRPO achieves superior token efficiency and establishes a new state-of-the-art 7B reasoning model with only 27 hours of compute."
1. Base Model Secrets: Pretraining Biases & Templates
The authors investigated several base models, including the Qwen2.5 family, Llama-3.1, and the massive DeepSeek-V3-Base (685B). They discovered two shocking facts:
- The Template Paradox: Qwen2.5-Math base models perform up to 60% better when NO template is used compared to standard 4-shot prompting. This suggests they were pretrained on raw, concatenated question-answer text, making them behave like SFT models out of the box.
- Pre-existing "Aha Moments": Pure base models, including DeepSeek-V3-Base, already naturally generate self-reflection keywords (like "Wait, let me recheck..." or "Aha!") at higher temperatures, even before any RL training.
Interactive: Template Impact Explorer
Toggle between templates and models to see how formatting choices dramatically alter accuracy. Notice how Qwen2.5-Math peaks with No Template!
2. The GRPO Length & Difficulty Bias
Standard GRPO (Group Relative Policy Optimization) calculates the advantage of a response relative to a group of sampled outputs:
However, the paper uncovers a massive flaw in how GRPO computes its policy objective:
The inclusion of the response-level length normalization term $1/|o_i|$ introduces an unintended Length Bias:
- For correct answers ($\hat{A}_{i,t} > 0$): Shorter responses get larger gradient updates. The model is pushed to be brief.
- For incorrect answers ($\hat{A}_{i,t} < 0$): Shorter incorrect responses are penalized *more* heavily, while longer incorrect responses are penalized *less* (due to the dividing factor $|o_i|$). This teaches the model that if it's going to fail, it should fail with a massive wall of text!
Interactive: GRPO Length Bias Simulator
Adjust the length of incorrect answers and see how standard GRPO fails to penalize long wrong answers, while Dr. GRPO maintains a fair penalty.
As incorrect length increases, the penalty per token shrinks, encouraging bloated wrong answers.
Removing the length term ensures incorrect answers are penalized consistently regardless of length.
3. Dr. GRPO: Group Relative Policy Optimization Done Right
To fix these biases, the authors propose a dead-simple yet mathematically rigorous solution: remove the normalization terms.
The Dr. GRPO Objective:
$$J_{\text{Dr. GRPO}}(\pi_\theta) = \mathbb{E} \left[ \frac{1}{G} \sum_{i=1}^G \sum_{t=1}^{|o_i|} \min \left( \frac{\pi_\theta(o_{i,t})}{\pi_{\theta_{\text{old}}}(o_{i,t})} \tilde{A}_{i}, \text{clip}(\dots)\tilde{A}_{i} \right) \right]$$By removing the standard deviation normalization, they also eliminate Question-level difficulty bias, which historically over-weighted extremely easy or extremely hard questions where the reward variance was tiny.
Interactive: The "Aha!" Moment Simulator
Simulate how different models generate self-reflection tokens at varying temperatures. Higher temperatures trigger exploratory paths where the model realizes mistakes!
Results & Benchmarks
By combining the unbiased Dr. GRPO with a minimalist recipe (tuning Qwen2.5-Math-7B on MATH levels 3-5), the authors achieved state-of-the-art reasoning performance with incredibly low compute overhead (only 27 hours on 8x A100 GPUs).
AIME 2024 Performance vs. Competitors
* Note: All models evaluated at a 3k token budget limit to ensure fair comparison.
Limitations & Open Questions
Pretraining Contamination?
The massive performance boost of Qwen2.5 models when evaluated *without* templates strongly suggests they were pretrained on high-quality SFT data. This makes it difficult to claim they are "pure" base models undergoing R1-style emergent learning.
The Compute Floor
While Dr. GRPO is incredibly efficient, reasoning capabilities still scale heavily with base model size. The gap between 1.5B, 7B, and 685B models remains massive, indicating that RL cannot fully compensate for raw parameter limitations.
Glossary
GRPO (Group Relative Policy Optimization)
An RL algorithm that estimates policy advantages by sampling a group of outputs for a single prompt, avoiding the need for a separate value model (critic), which saves GPU memory.
Aha Moment / Self-Reflection
An emergent behavior where a language model halts its current trajectory, recognizes a logical error, and restarts with a corrected approach (e.g., "Wait, that's wrong. Let me re-evaluate...").
RLOO (REINFORCE Leave-One-Out)
An unbiased baseline estimator for policy gradients. The authors prove that Dr. GRPO is mathematically equivalent to RLOO up to a scaling factor.
Cite this Paper
@inproceedings{liu2025understanding,
title={Understanding R1-Zero-Like Training: A Critical Perspective},
author={Liu, Zichen and Chen, Changyu and Li, Wenjun and Qi, Penghui and Pang, Tianyu and Du, Chao and Lee, Sun Wee and Lin, Min},
booktitle={Conference on Language Modeling (COLM)},
year={2025}
}