← Flash PapersDAPO: Demystifying Large-Scale LLM Reinforcement Learning

Share this Flash Paper

DAPO: Demystifying Large-Scale LLM Reinforcement Learning

https://flashpapers.ai/p/vztjrtkjz6
Make your own
Qiying Yu, Zheng Zhang +33 more
Details ▸
AIME 2024 Accuracy:50.0%· AIME 2024Training Step Efficiency:50% fewer steps· AIME 2024
Abstract

Inference scaling empowers LLMs with unprecedented reasoning ability, with reinforcement learning as the core technique to elicit complex reasoning. However, key technical details of state-of-the-art reasoning LLMs are concealed (such as in OpenAI o1 blog and DeepSeek R1 technical report), thus the community still struggles to reproduce their RL training results. We propose the $\textbf{D}$ecoupled Clip and $\textbf{D}$ynamic s$\textbf{A}$mpling $\textbf{P}$olicy $\textbf{O}$ptimization ($\textbf{DAPO}$) algorithm, and fully open-source a state-of-the-art large-scale RL system that achieves 50 points on AIME 2024 using Qwen2.5-32B base model. Unlike previous works that withhold training details, we introduce four key techniques of our algorithm that make large-scale LLM RL a success. In addition, we open-source our training code, which is built on the verl framework, along with a carefully curated and processed dataset. These components of our open-source system enhance reproducibility and support future research in large-scale LLM RL.