1 Why This Matters
Historically, training Large Language Models to reason (like solving complex math or coding problems) required vast amounts of high-quality, human-annotated reasoning trajectories. This paradigm has two major bottlenecks:
- Human Trajectory Scarcity: Creating step-by-step solutions for thousands of complex math Olympiad or competitive programming problems is incredibly expensive and slow.
- Cognitive Ceiling: By training models strictly to replicate human thinking, we inadvertently cap their intelligence at human levels and introduce human cognitive biases.
DeepSeek-R1 changes this. It demonstrates that highly complex reasoning capabilities—such as self-correction, backtracking, and verification—can emerge organically through pure reinforcement learning without any prior supervised fine-tuning.
2 The Big Idea
The authors developed two core models: DeepSeek-R1-Zero and DeepSeek-R1.
DeepSeek-R1-Zero
Trained via pure RL on top of a base model without any initial Supervised Fine-Tuning (SFT). It achieved astonishing reasoning abilities but suffered from poor readability, chaotic language mixing, and repetitive loops.
DeepSeek-R1
A multi-stage pipeline that starts with a tiny "cold-start" SFT dataset to ensure clean formatting, followed by RL, rejection sampling, and a secondary SFT stage. It achieves top-tier reasoning while remaining aligned, highly readable, and multilingual.
3 Interactive: GRPO vs PPO
To train at scale, DeepSeek uses Group Relative Policy Optimization (GRPO) instead of traditional Proximal Policy Optimization (PPO). GRPO completely eliminates the memory-heavy Value Model (Critic) by estimating baseline advantages directly from a group of sampled outputs.
Architecture Memory Footprint
(Actor)
(Critic)
(Actor)
$\text{mean}(r), \text{std}(r)$
GRPO Advantage Formulation
Instead of training an expensive neural network to predict expected rewards, GRPO samples a group of outputs ($G=16$) and computes relative performance directly.
4 How It Works: The Multi-Stage Pipeline
To solve the usability problems of pure RL (DeepSeek-R1-Zero), the final DeepSeek-R1 model uses a highly optimized four-stage training pipeline:
Cold-Start SFT
Fine-tuning DeepSeek-V3-Base on a tiny dataset of thousands of high-quality, long-chain-of-thought (CoT) data. This seeds the model with clean formatting (enclosed in <think>...</think> tags) and prevents initial chaos.
Reasoning-Focused RL
Large-scale RL focused on mathematics, coding, and logic. Rewards are strictly rule-based (e.g., compiler feedback, exact match math solvers) plus a Language Consistency Reward to prevent language mixing.
Rejection Sampling & SFT
Generating 800,000 training samples from the Stage 2 checkpoint. Only correct, highly readable solutions are kept (600k reasoning + 200k general helpfulness/harmlessness data) to train a new, highly robust model.
Preference Alignment RL
The final RL stage using helpfulness and harmlessness reward models alongside accuracy rewards to align model behavior with human preferences.
5 Interactive: The "Aha Moment"
During reinforcement learning, the authors witnessed a fascinating emergent behavior: the model learned to use anthropomorphic phrases like "Wait, wait..." to pause, reflect, and correct its own mistakes. Step through a simplified simulation of this phenomenon below.
<think>
Let's set $S_n = m$, where $m$ is an integer.
Then we have: $m^2 = n^4 + 289 \implies m^2 - n^4 = 289$.
6 Interactive: Distillation Explorer
One of the paper's most critical discoveries is that distilling DeepSeek-R1's reasoning outputs into smaller models (like Qwen-1.5B or Llama-8B) yields far superior performance than training those smaller models using RL from scratch. Select a model below to compare its benchmark scores against industry giants.
7 Limitations & Open Questions
While DeepSeek-R1 marks a massive leap forward, the paper candidly addresses several critical limitations:
- Suboptimal Structured Output & Tool Use: DeepSeek-R1 currently struggles with highly formatted outputs (like JSON schemas) and cannot leverage external tools like search engines or calculators.
- Token Efficiency & Overthinking: The model dynamically scales compute at test-time, but sometimes "overthinks" simple questions, spending thousands of unnecessary tokens on trivial problems.
- Language Mixing: When queried in languages other than English or Chinese, the model occasionally performs its internal reasoning in a mix of English and Chinese, degrading readability.
- Prompt Sensitivity: DeepSeek-R1 is highly sensitive to system prompts. Few-shot prompts consistently degrade its reasoning performance compared to a pure zero-shot setting.
8 Glossary
GRPO (Group Relative Policy Optimization)
A reinforcement learning algorithm that optimizes language models without training an auxiliary value model (critic). It samples a group of outputs for each prompt and estimates advantages based on the relative rewards within the group, saving massive amounts of VRAM.
CoT (Chain of Thought)
A prompting and training technique where the model generates intermediate reasoning steps before producing the final answer. This dramatically improves performance on complex reasoning, coding, and mathematical tasks.
Model Distillation
The process of transferring knowledge from a large, high-performing model (the teacher) to a smaller, more efficient model (the student) by training the student on the teacher's outputs.
9 Citation
@misc{deepseek2025r1,
title={DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning},
author={DeepSeek-AI},
year={2025},
eprint={2501.12948},
archivePrefix={arXiv},
primaryClass={cs.CL}
}