arXiv:2402.03300

DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, et al. • DeepSeek-AI, Tsinghua, Peking University • 2024

TL;DR

DeepSeekMath 7B achieves an impressive 51.7% on the competition-level MATH benchmark without external tools, approaching GPT-4 performance. It succeeds through a massive, high-quality 120B token multilingual mathematical corpus and a novel reinforcement learning algorithm, Group Relative Policy Optimization (GRPO), which removes the need for a separate critic model to dramatically save memory.

120B Math Corpus GRPO RL Algorithm 51.7% MATH Top-1 Critic-Free PPO

Why This Matters

Mathematical reasoning has long been the holy grail of LLM capabilities. Unlike natural language generation, math requires precise, multi-step logical deduction where a single error invalidates the entire output. Closed-source giants like GPT-4 and Gemini-Ultra have dominated these tasks, while open-source models lagged significantly.

DeepSeekMath bridges this gap. It proves that with meticulous data curation and efficient reinforcement learning, a relatively small 7B parameter model can match or exceed closed-source models that are orders of magnitude larger.

The Big Idea

The mathematical reasoning capability of DeepSeekMath is attributed to two key pillars:

Iterative Web Data Mining: A highly engineered pipeline that extracted 120B high-quality math tokens from Common Crawl, bypassing the limits of standard math pre-training sets.
Group Relative Policy Optimization (GRPO): A reinforcement learning algorithm that eliminates the heavy critic model of standard PPO. Instead, it computes advantages relatively within a group of sampled outputs, reducing training memory by nearly half and enabling efficient training at scale.

1. Math Pre-Training at Scale

How DeepSeek built a 120B token math corpus from raw web pages.

Standard web crawls contain vast amounts of noisy data. To extract high-quality mathematical content, DeepSeek developed an iterative fastText-based data selection pipeline. They started with a high-quality seed corpus (OpenWebMath) and progressively trained classifiers to mine more mathematical domains from 40 billion HTML pages.

Interactive Pipeline Explorer

Click steps to simulate iterations

Iteration 1: Setting up the Seed

We initialize the classifier using OpenWebMath (13.6B tokens) as positive training examples and random web pages as negative examples.

Corpus Growth 13.6B tokens

2. Group Relative Policy Optimization (GRPO)

Eliminating the critic to optimize reinforcement learning memory.

Traditional Proximal Policy Optimization (PPO) requires a critic model of equivalent size to the policy model to estimate the baseline value. This significantly increases GPU memory demands.

GRPO eliminates the critic model. For each prompt $q$, it samples a group of $G$ outputs $\{o_1, o_2, \dots, o_G\}$ from the policy model. It then computes the reward for each output and normalizes them within the group to get the advantage:

$$\hat{A}_{i,t} = \frac{r_i - \text{mean}(\mathbf{r})}{\text{std}(\mathbf{r})}$$

This relative advantage aligns perfectly with the comparative nature of reward models, while reducing training memory overhead by up to 43%.

Memory & Architecture Cost Simulator

Model Size:

Standard PPO

196 GB

Actor Model (Trainable) 84 GB

Critic Model (Trainable) 84 GB

Ref & Reward (Frozen) 28 GB

DeepSeek GRPO

112 GB

Actor Model (Trainable) 84 GB

Critic Model Removed

Ref & Reward (Frozen) 28 GB

GRPO reduces active training memory requirements by 42.8%, allowing you to train larger models on the same hardware.

3. The Unified RL Paradigm

Understanding alignment algorithms through a single mathematical framework.

DeepSeekMath presents a beautiful unifying theory: all major alignment methods (SFT, RFT, DPO, PPO, and GRPO) can be conceptualized as variants of a single reinforcement learning objective. They only differ in three factors: Data Source, Reward Function, and Gradient Coefficient.

The generalized gradient can be written as:

$$\nabla_\theta J_{\mathcal{A}}(\theta) = \mathbb{E}_{(q,o) \sim \mathcal{D}} \left[ \frac{1}{|o|} \sum_{t=1}^{|o|} GC_{\mathcal{A}}(q, o, t, \pi_{rf}) \nabla_\theta \log \pi_\theta(o_t | q, o_{

Unified Paradigm Playground

Select algorithm to inspect

Data Source

Human selected SFT data

Reward Function

None (Implicitly Human)

Gradient Coefficient (GC)

Live Math Update Simulation

Prompt: "Solve $x^2 - 5x + 6 = 0$"

Performance Results

How DeepSeekMath stacks up against the world's best models on the MATH benchmark.

MATH Benchmark Accuracy (Top-1)

Note: All scores represent Top-1 accuracy on the competition-level MATH benchmark without external tools.

Limitations & Open Questions

Despite the incredible performance of DeepSeekMath, the authors highlight several remaining hurdles:

Geometry & Formal Proofs: The model's performance remains relatively weak on geometry tasks (e.g., handling triangles and ellipses), indicating a possible selection bias in the pre-training corpus.
Few-Shot Scaling: Unlike GPT-4, which improves significantly with few-shot prompting, DeepSeekMath shows similar performance in zero-shot and few-shot settings, limited by its 7B parameter scale.
Noisy Reward Signals: The pipeline relies heavily on the reward model's accuracy. Even carefully annotated math datasets (like PRM800K) contain up to 20% incorrect annotations, pointing to the need for algorithms robust to noisy rewards.

Glossary

OpenWebMath

A high-quality, open-source dataset of mathematical web pages extracted from Common Crawl, used as the initial seed dataset for DeepSeekMath's iterative pipeline.

Critic Model

In standard Actor-Critic RL (like PPO), the critic is a separate neural network trained to predict the expected return (value) of a state, helping baseline the policy's updates.

Self-Consistency

A decoding strategy where the model generates multiple reasoning paths, and the final answer is selected via a majority vote among the generated outputs.

Citation

@misc{shao2024deepseekmath,
  title={DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models}, 
  author={Zhihong Shao and Peiyi Wang and Qihao Zhu and Runxin Xu and Junxiao Song and Xiao Bi and Haowei Zhang and Mingchuan Zhang and Y.K. Li and Y. Wu and Daya Guo},
  year={2024},
  eprint={2402.03300},
  archivePrefix={arXiv},
  primaryClass={cs.CL}
}