DAPO: Demystifying Large-Scale LLM Reinforcement Learning

Why This Matters

We are in the era of test-time scaling. Models like OpenAI's o1 and DeepSeek's R1 demonstrate that allowing LLMs to "think" longer via a Chain-of-Thought (CoT) dramatically increases performance on complex mathematical and coding tasks.

The core driver of this reasoning capability is large-scale Reinforcement Learning (RL). However, training models to think is notoriously unstable. Standard algorithms like GRPO suffer from entropy collapse (where the model becomes overly deterministic too early, stopping exploration), reward noise, and extreme gradient variance. Without proprietary, closely guarded training recipes, open-source attempts often stall at mediocre performance levels.

The Big Idea

DAPO (Decoupled Clip and Dynamic Sampling Policy Optimization) is an open-source, highly scalable RL framework built on top of the verl library. It introduces four critical engineering and algorithmic modifications to standard GRPO, allowing a 32-billion parameter base model to achieve state-of-the-art competitive math reasoning without needing complex proprietary pipelines.

The DAPO Objective

$$ \mathcal{J}_{\text{DAPO}}(\theta) = \mathbb{E}_{(q,a)\sim\mathcal{D}, \{o_i\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}(\cdot|q)} \left[ \frac{1}{\sum_{i=1}^G |o_i|} \sum_{i=1}^G \sum_{t=1}^{|o_i|} \min \left( r_{i,t}(\theta)\hat{A}_{i,t}, \text{clip}(r_{i,t}(\theta), 1 - \varepsilon_{\text{low}}, 1 + \varepsilon_{\text{high}})\hat{A}_{i,t} \right) \right] $$

Notice the decoupled clipping bounds ($\varepsilon_{\text{low}}, \varepsilon_{\text{high}}$) and the token-level double summation.

How It Works: The 4 Key Innovations

1. Raise the Ceiling: Clip-Higher

In standard PPO/GRPO, the trust region is restricted symmetrically (e.g., $\varepsilon = 0.2$). This prevents the policy ratio $r_t(\theta)$ from exceeding $1.2$ or dropping below $0.8$.

However, this symmetric bound creates an asymmetric penalty in practice. For an "exploratory" token with a low initial probability (e.g., $P_{\text{old}} = 0.01$), the maximum allowed probability update is bounded at $0.012$. Conversely, a high-probability "exploitation" token (e.g., $P_{\text{old}} = 0.9$) can easily scale up. This dramatically restricts the exploration of low-probability but correct paths, leading to entropy collapse.

The Solution: Decouple the bounds. Keep $\varepsilon_{\text{low}} = 0.2$ to prevent catastrophic policy drift, but raise $\varepsilon_{\text{high}} = 0.28$ to allow promising low-probability exploration tokens to scale up their probabilities faster.

Interactive: Clip-Higher Explorer

Adjust the upper clipping limit ($\varepsilon_{\text{high}}$) to see how it unlocks probability updates for low-probability "exploration" tokens compared to high-probability "exploitation" tokens.

Upper Clip ($\varepsilon_{\text{high}}$): 0.28

0.10 (Strict) 0.20 (Standard) 0.50 (Loose)

Exploration Token ($P_{\text{old}} = 1\%$): 1.28%

Exploitation Token ($P_{\text{old}} = 80\%$): 102.40%

Exploration Token Probability Boost +28%

Exploitation Token Probability Boost +28%

By raising the ceiling, low-probability paths (crucial for complex math reasoning) get enough gradient push to be learned before the policy collapses into a deterministic state.

2. Dynamic Sampling

In group-based RL (like GRPO), we sample $G$ outputs for each prompt. If a prompt's sampled responses are all correct (all $1$s) or all incorrect (all $0$s), the relative advantage for every response in that group becomes exactly zero: $$ \hat{A}_{i,t} = \frac{R_i - \text{mean}(R)}{\text{std}(R)} \implies 0 $$

Zero advantage means zero policy gradient. As the model improves, more and more prompts achieve 100% success in their rollouts. This wastes massive computational resources on zero-gradient steps and increases gradient noise.

The Solution: DAPO over-samples and dynamically filters out prompts where the accuracy is strictly $1$ or $0$. The batch is continuously filled with active-learning prompts (where there is a mix of correct and incorrect answers) before computing the gradient updates.

Interactive: Dynamic Sampling Visualizer

Toggle between standard GRPO and DAPO's Dynamic Sampling to see how dead/zero-gradient batches are filtered out to keep training highly efficient.

Active Gradient Batches: 2 / 4

Wasted Compute (Zero Gradient): 50%

3. Token-Level Policy Gradient Loss

Standard GRPO averages loss at the sample level. Every response, regardless of length, contributes equally to the final gradient.

In long-CoT reasoning, this creates two major issues:

Tokens in longer, high-quality reasoning paths have a disproportionately lower individual contribution to the loss.
The model is not penalized enough for generating excessively long, repetitive, or "gibberish" sequences.

The Solution: DAPO switches to a token-level loss, where the double summation over both samples and tokens is normalized by the total number of tokens in the entire batch, giving longer sequences their fair share of influence.

Interactive: Loss Weighting Calculator

Adjust the length of a correct, long-reasoning response to see how its influence on the gradient updates changes under Sample-Level vs. Token-Level loss formulations.

Response 1 (Short, Correct) Length: 100 tokens

Response 2 (Long Reasoning) Length: 500 tokens

Sample-Level Loss (Equal Weight)

Response 1

50.0%

Response 2

50.0%

Token-Level Loss (Proportional Weight)

Response 1

16.7%

Response 2

83.3%

4. Hide and Seek: Overlong Reward Shaping

When responses exceed the maximum generation token limit ($L_{\text{max}}$), they are truncated. Simply giving truncated responses a hard penalty (e.g., $-1$) introduces severe noise, as a correct and highly logical reasoning path might be penalized solely because it ran out of tokens.

The Solution: DAPO introduces Soft Overlong Punishment. If the response length $|y|$ falls within a buffer zone ($L_{\text{max}} - L_{\text{cache}}$), it receives a smooth, length-proportional penalty rather than a cliff-edge drop.

$$ R_{\text{length}}(y) = \begin{cases} 0, & |y| \le L_{\text{max}} - L_{\text{cache}} \\ \frac{(L_{\text{max}} - L_{\text{cache}}) - |y|}{L_{\text{cache}}}, & L_{\text{max}} - L_{\text{cache}} < |y| \le L_{\text{max}} \\ -1, & L_{\text{max}} < |y| \end{cases} $$

Headline Results & Ablation

By progressively applying these techniques, the authors demonstrate a massive step-by-step improvement on the AIME 2024 benchmark using the Qwen2.5-32B base model.

Ablation Study: Progressive Accuracy on AIME 2024

Naive GRPO Baseline 30%

+ Overlong Filtering 36%

+ Clip-Higher 38%

+ Soft Overlong Punishment 41%

+ Token-level Loss 42%

+ Dynamic Sampling (DAPO Full) 50%

SOTA Open-Source LLM RL

Limitations & Open Questions

Overfitting to the Training Set: The authors note that final rewards on the training set often show little correlation with validation accuracy, highlighting a risk of overfitting to specific rule-based reward structures.
Task Scope: Currently, evaluations are focused heavily on mathematical reasoning (AIME). How well these exact hyperparameter configurations transfer to creative writing or unstructured instruction-following remains an open question.
Dynamic Sampling Cost: While Dynamic Sampling reduces gradient steps, it requires continuous rollout sampling until the active criteria are met, shifting some compute burden from backpropagation to generation.

Glossary

Entropy Collapse

A phenomenon where the policy becomes highly deterministic, focusing solely on a few high-probability actions and losing its ability to explore alternative solutions.

GRPO (Group Relative Policy Optimization)

An RL framework that removes the need for an explicit critic/value network by estimating advantage values relative to a group of sampled candidate responses for the same input prompt.

Chain-of-Thought (CoT)

Generating intermediate reasoning steps before producing the final output, which has been shown to significantly boost performance in complex logical tasks.

Citation

@article{yu2025dapo,
  title={DAPO: An Open-Source LLM Reinforcement Learning System at Scale},
  author={Yu, Qiying and Zhang, Zheng and Zhu, Ruofei and Yuan, Yufeng and Zuo, Xiaochen and Yue, Yu and others},
  journal={arXiv preprint arXiv:2503.14476},
  year={2025}
}

DAPO: Decoupled Clip & Dynamic Sampling

Headline Results