FlashPapers
Preprint • Feb 2026

Spurious Rewards: Rethinking Training Signals in RLVR

Rulin Shao*, Shuyue Stella Li*, Rui Xin*, Scott Geng*, Yiping Wang, Sewoong Oh, Simon Shaolei Du, Nathan Lambert, Sewon Min, Ranjay Krishna, Yulia Tsvetkov, Hannaneh Hajishirzi, Pang Wei Koh, Luke Zettlemoyer

TL;DR (Too Long; Didn't Read)

Reinforcement Learning with Verifiable Rewards (RLVR) can dramatically boost math reasoning performance even when trained with completely random or incorrect rewards. This counterintuitive phenomenon is driven by GRPO's clipping mechanism, which acts as an exploitation bias that amplifies pre-existing latent priors (like writing Python code blocks) rather than teaching new capabilities.

Random Rewards Work GRPO Clipping Bias Code Reasoning Prior Model-Dependent Mirage

Why This Matters

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as the premier paradigm for scaling LLM reasoning, as seen in models like DeepSeek-R1. By providing binary rewards for mathematically verifiable answers, models learn to output structured, long-form chains of thought. However, the exact mechanics of these gains remain poorly understood.

This paper uncovers a startling, counterintuitive reality: the reward signal itself might not be what's driving the learning. On popular math models like Qwen2.5-Math-7B, training with completely random noise (Bernoulli rewards with no relation to the prompt) or even rewarding incorrect answers yields up to 21.4% absolute accuracy gains on MATH-500, nearly matching the 29.1% gain of ground-truth rewards.

This suggests that much of the RLVR literature on open-source models may be observing a mirage—an amplification of pre-existing, latent capabilities rather than the acquisition of new reasoning skills.

The Big Idea

The core hypothesis is the Elicitation Hypothesis: RLVR training at current compute scales does not teach models new reasoning capabilities. Instead, the optimization process acts as an activation key for latent, high-performing behaviors already present in the pre-training distribution. GRPO's clipping mechanism introduces an algorithmic bias that systematically reinforces these high-prior behaviors, even when the feedback loop is entirely noisy or spurious.

How It Works

1. GRPO and the Clipping Bias

Group Relative Policy Optimization (GRPO) avoids a critic network by normalizing rewards within a group of outputs sampled from the same prompt. The objective function with clipping is defined as:

$$J(\theta) = \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_{\text{old}}} \left[ \sum_{t=1}^{|y|} \min \left( \rho_t(y; \theta) \hat{A}(x, y), \text{clip}(\rho_t(y; \theta), 1 - \epsilon_c, 1 + \epsilon_c) \hat{A}(x, y) \right) \right]$$

Where $\rho_t(y; \theta) = \frac{\pi_\theta(y_t | x, y_{

When rewards are random, the expected advantage $\mathbb{E}[\hat{A}] = 0$. Without clipping, this would yield zero expected gradient update. However, the inclusion of the clipping term introduces an asymmetric bias.

Widget 1: GRPO Clipping Bias Simulator

Interactive

Adjust the pre-training token probability ($\pi_{\text{old}}$) and clipping threshold ($\epsilon_c$) to visualize how GRPO's clipping bounds create asymmetric gradient updates under uninformative (random) rewards.

Low Prior (Rare) High Prior (Frequent)
Analysis

Calculating...

Absolute Clipping Range
Clipping Range
Prior
0.0 (Impossible) 1.0 (Certainty)
Upper limit ($\pi_{\text{old}} \cdot (1+\epsilon_c)$): 1.02
Gradient Bias: Positive

2. The Case Study: Code Reasoning

For Qwen2.5-Math-7B, the dominant pre-existing behavior is code reasoning: writing Python code blocks to structure math problems, and then autoregressively predicting the execution output without actually calling an external interpreter.

This behavior is highly predictive of accuracy even before RL training: answers containing code have an accuracy of 60.9%, compared to 35.0% for natural language answers.

Widget 2: Strategy Transition & Performance Contribution

Interactive

Select a model to see how its reasoning strategies shifted after RLVR training, and the partial contribution of each transition to the overall performance gain.

Lang → Code 58.3%

The model begins with natural language but learns to output Python. This transition represents the single largest driver of accuracy gains.

Lang → Lang 21.4%

The model stays in natural language format. Represents standard reasoning improvement without formatting shifts.

Code → Code 11.6%

The model wrote code before and continues to do so, with slight optimization in code quality or execution correctness.

Interactive Playground

To demystify how these spurious signals interact with different model architectures, use the matrix below. Toggle between different pre-trained models and reward functions to see the resulting MATH-500 performance.

Widget 3: Model & Reward Signal Matrix

Interactive
Simulated Outcome
MATH-500 Accuracy
78.5% +29.1%
Baseline: 49.4%

Ground truth rewards provide strong mathematical signals, guiding the model to optimal performance.

Prior Amplification: High

Results & Generalization

The findings challenge the generalization of RLVR research. While Qwen models show dramatic gains from spurious rewards, other model families (like Llama and OLMo) do not. This divergence suggests that pre-training distributions dictate RL dynamics far more than the RL algorithm itself.

Model Baseline Ground Truth Random Reward Incorrect Reward
Qwen2.5-Math-7B 49.4% 78.5% (+29.1%) 70.8% (+21.4%) 73.5% (+24.1%)
Qwen2.5-7B 41.6% 76.0% (+34.4%) 58.0% (+16.4%) 54.8% (+13.2%)
Llama3.1-8B-Instruct 36.8% 44.2% (+7.4%) 35.5% (-1.3%) 30.5% (-6.3%)
OLMo2-7B 9.0% 24.5% (+15.5%) 0.7% (-8.3%) 2.6% (-6.4%)

Limitations & Open Questions

  • AIME 2025 Hard Cutoff: The performance gains from spurious rewards completely vanish on the AIME 2025 benchmark. This suggests that spurious rewards cannot elicit knowledge that is completely out-of-distribution or beyond the model's pre-trained threshold.
  • Model-Centric Bias: The RLVR literature is heavily over-indexed on Qwen models. Because Qwen possesses highly structured, latent "code reasoning" behaviors, it is unusually robust to uninformative rewards. Other architectures fail to replicate these gains.
  • The Role of Entropy: The study disables KL divergence and entropy regularization to isolate the clipping bias. The exact interaction of clipping bias with dynamic KL penalties remains an open area of study.

Glossary

RLVR

Reinforcement Learning with Verifiable Rewards. A training setup where the model receives feedback based on the objective correctness of its final answer (e.g., matching a ground truth math value).

GRPO

Group Relative Policy Optimization. An RL algorithm that normalizes rewards within a group of outputs generated from the same prompt, eliminating the need for a separate critic network.

Clipping Bias

An optimization artifact where the policy update is asymmetrically bounded, leading to the positive reinforcement of high-probability tokens even under noisy or uninformative training signals.

Code Reasoning

The behavior where a language model autoregressively writes Python code and predicts its output to solve a math problem, without actually executing the code on an external environment.

Citation 

@article{shao2026spurious,
  title={Spurious Rewards: Rethinking Training Signals in RLVR},
  author={Shao, Rulin and Li, Shuyue Stella and Xin, Rui and Geng, Scott and Wang, Yiping and Oh, Sewoong and Du, Simon Shaolei and Lambert, Nathan and Min, Sewon and Krishna, Ranjay and Tsvetkov, Yulia and Hajishirzi, Hannaneh and Koh, Pang Wei and Zettlemoyer, Luke},
  journal={arXiv preprint arXiv:2506.10947},
  year={2026}
}
Made with Flash Papers — make your own ↗