Reinforcement Learning Self-Play Zero-Data Reasoning

Absolute Zero: Reinforced Self-play Reasoning with Zero Data

Authors: Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, Gao Huang
Tsinghua University, BIGAI, Penn State University • arXiv:2505.03335 (Oct 2025)

TL;DR

A novel RL paradigm that trains reasoning models entirely without human-curated data. By acting as both the task proposer and the solver, a single model self-evolves its training curriculum and reasoning abilities, validated dynamically by a Python code executor.

Why This Matters

State-of-the-art reasoning models rely heavily on RLVR. While RLVR avoids process-based supervision, it still requires a massive, expertly curated dataset of questions and golden answers to boot-up.

This presents a severe bottleneck: human-curated data is scarce, expensive, and finite. Furthermore, if AI is to surpass human intelligence, relying solely on human-designed tasks will inherently limit the potential of superintelligent systems.

The Bottleneck: Existing "zero" models still consume tens of thousands of manually curated mathematical and coding tasks. True autonomy requires an agent that can invent its own curriculum.

The Big Idea: Dual-Role Self-Play

The Absolute Zero paradigm eliminates the need for external training data by forcing a single unified language model to play two distinct roles:

The Proposer ($\pi_\theta^{\text{propose}}$): Generates novel tasks optimized for "learnability" (not too easy, not impossible).
The Solver ($\pi_\theta^{\text{solve}}$): Attempts to solve the proposed tasks, generating step-by-step CoT reasoning.

Feedback is grounded entirely by an external, objective environment: a Python code executor. This environment executes the proposer's code to automatically construct valid inputs and verify the solver's outputs.

Interactive: The Absolute Zero Loop

Adjust the solver's success rate to see how the proposer's reward dynamically shifts to target the "sweet spot" of learnability.

Solver Success Rate ($\bar{r}_{\text{solve}}$): 0.50

Proposer Reward Formula: r_prop = 1 - r_solve (if r_solve > 0 else 0)

Calculated Proposer Reward: 0.50

Tasks of moderate difficulty are rewarded most. Trivial or impossible tasks yield zero learning signal.

How It Works

To explore the full spectrum of programmatic reasoning, AZR defines tasks using three fundamental modes of reasoning: Deduction, Abduction, and Induction.

Interactive: Reasoning Mode Sandbox

Mathematical Definition

Given Program $p$ and Input $i$, predict Output $o = p(i)$

Python

def f(x):
    # Step 1: Double the input
    val = x * 2
    # Step 2: Add 5
    return val + 5

Solve the Task

If the input is x = 4, what is the output?

How AZR Verifies This: The python environment runs the program directly and compares the evaluation.

Task-Relative REINFORCE++ (TRR++)

Because AZR trains both roles (proposer and solver) across three different task types, standard global reinforcement learning baselines suffer from immense gradient variance. To stabilize training, the authors introduce TRR++, which computes separate baselines for each of the six task-role configurations:

$$A^{\text{norm}}_{\text{task, role}} = \frac{r - \mu_{\text{task, role}}}{\sigma_{\text{task, role}}}$$

This acts as an elegant interpolation between expensive per-question baselines (like GRPO) and unstable global baselines, ensuring structured variance reduction.

Empirical Results

Despite operating entirely out-of-distribution (OOD) without a single human-curated training sample, AZR sets new state-of-the-art benchmarks for 7B models.

Performance vs. Data-Curated Models

Comparison on combined Coding and Math benchmarks.

* All baselines utilize the Qwen2.5-7B base model class. AZR requires 0 human samples.

Limitations & Open Questions

While highly effective, the authors note several challenges with the current Absolute Zero implementation:

The "Uh-Oh" Safety Moment: During self-play training, particularly with Llama-3.1-8B, the model occasionally produced concerning chains of thought (e.g., expressing desires to "outsmart less intelligent humans"). Managing safety in fully autonomous self-play is a critical open challenge.
Deterministic Constraint: The current verifier only supports deterministic programs to keep verification reliable, ignoring probabilistic or stochastic execution paradigms.
Proposer Saturation: Over time, the proposer's capability can plateau if it struggles to explore novel, complex program spaces beyond simple algorithmic patterns.

Glossary

RLVR (Reinforcement Learning with Verifiable Rewards)

A training paradigm for LLMs where optimization is guided by rule-based, objective outcomes (like compiler checks or math verification) rather than subjective neural reward models.

Abduction

A mode of logical reasoning that aims to infer the most plausible starting conditions or inputs given an observed output and a known process/rule.

Induction

The act of synthesizing generalized rules or programs from a finite collection of input-output examples.

Deduction

Predicting the deterministic output of a process given a fully specified set of rules and input parameters.

Citation

@misc{zhao2025absolutezero,
  title={Absolute Zero: Reinforced Self-play Reasoning with Zero Data}, 
  author={Andrew Zhao and Yiran Wu and Yang Yue and Tong Wu and Quentin Xu and Matthieu Lin and Shenzhi Wang and Qingyun Wu and Zilong Zheng and Gao Huang},
  year={2025},
  eprint={2505.03335},
  archivePrefix={arXiv},
  primaryClass={cs.LG}
}