Reinforcement Learning Alignment & Reasoning Alias: RLVR

Reinforcement Learning with Verifiable Rewards

A training paradigm for language models where alignment and reasoning capabilities are optimized using objective, programmatically verifiable feedback (such as unit tests or mathematical equivalence) rather than subjective human preferences or noisy neural reward models.

In One Sentence

RLVR bypasses the instability and exploitability of traditional reward models by leveraging deterministic oracles—like compilers, interpreters, or symbolic math solvers—to provide clean, binary, or structured rewards directly to the policy.

Intuition

In standard RL from Human Feedback (RLHF), we train a neural Reward Model (RM) to mimic human preferences. While effective for style and safety, these RMs are notoriously prone to reward hacking—where the generator learns to output verbose, sycophantic, or superficially pleasing text that tricks the RM without actually solving the underlying problem.

For reasoning tasks like mathematics, coding, and formal logic, we do not need to guess if an answer is correct. We can verify it. If a model outputs a Python script, we can run it against unit tests. If it solves a calculus problem, we can parse the final expression and check mathematical equivalence.

By substituting the neural reward model with a deterministic Verifier, RLVR provides an un-hackable ground-truth signal. Under this regime, models are incentivized to develop robust internal reasoning, self-correction, and verification strategies (like chain-of-thought) because only the final correct execution yields a reward.

Emergent Behaviors: Recent breakthroughs (e.g., DeepSeek-R1) demonstrate that pure RLVR over long context lengths causes models to spontaneously learn self-reflection, error correction, and alternative path exploration without any human-annotated reasoning demonstrations.

Formal Definition

Let $x \in \mathcal{X}$ be a prompt (e.g., a math problem or coding prompt), and $y = (y_1, y_2, \dots, y_T) \in \mathcal{Y}$ be the generated sequence of tokens containing both the reasoning steps (Chain-of-Thought) and the final answer.

Verifiable Reward Function

The reward $R(x, y)$ is computed via a deterministic verification function $V$: $$R(x, y) = V(\text{extract}(y), y^*)$$ where $\text{extract}(y)$ isolates the final answer or executable code block from the generation, and $y^*$ is the ground-truth solution or validation test suite. Typically, $R(x, y) \in \{0, 1\}$ or is structured as a step-wise execution score.

To optimize the policy $\pi_\theta$ without the memory overhead of a separate critic network, RLVR often employs Group Relative Policy Optimization (GRPO). For a given prompt $x$, we sample a group of $G$ outputs $\{y^{(1)}, y^{(2)}, \dots, y^{(G)}\}$ from the current policy $\pi_\theta$.

The advantage $A_i$ for each candidate output $y^{(i)}$ is computed relative to its peers in the group: $$A_i = \frac{R(x, y^{(i)}) - \mu}{\sigma}$$ where $\mu = \frac{1}{G}\sum_{j=1}^G R(x, y^{(j)})$ and $\sigma = \sqrt{\frac{1}{G}\sum_{j=1}^G (R(x, y^{(j)}) - \mu)^2 + \epsilon}$.

The policy is then updated by maximizing the objective: $$\mathcal{L}_{\text{GRPO}}(\theta) = \frac{1}{G} \sum_{i=1}^G \left[ \min \left( r_i(\theta) A_i, \text{clip}(r_i(\theta), 1-\epsilon, 1+\epsilon) A_i \right) - \beta D_{\text{KL}}(\pi_\theta || \pi_{\text{ref}}) \right]$$ where $r_i(\theta) = \frac{\pi_\theta(y^{(i)} | x)}{\pi_{\theta_{\text{old}}}(y^{(i)} | x)}$ and $\pi_{\text{ref}}$ is the reference policy (usually the initial SFT model) to prevent policy collapse.

Interactive Walkthrough

Simulate how Group Relative Policy Optimization (GRPO) computes advantages for verifiable rewards.

1. Set Verifier Outputs

Toggle whether each of the 5 sampled model responses passed the deterministic verifier ($R=1$) or failed ($R=0$).

Sample 1 (Correct)

Sample 2 (Correct)

Sample 3 (Incorrect)

Sample 4 (Incorrect)

Sample 5 (Incorrect)

2. Group Advantage Calculation

Group Mean ($\mu$) 0.40

Group Std Dev ($\sigma$) 0.49

Relative Advantages ($A_i$):

Notice how samples that score above the group mean receive positive advantages, while those below receive negative advantages.

If all samples get the same reward (e.g., all correct or all incorrect), the standard deviation $\sigma$ approaches zero, and advantages are neutralized to prevent noisy updates.

Trade-offs & Practical Considerations

When to Use

• Deterministic Domains: Mathematics, computer programming (compilation + unit tests), and symbolic logic where correctness is absolute.
• Long Chain-of-Thought: When you want to incentivize models to "think" longer, self-correct, and try multiple paths before outputting a final answer.
• Resource-Constrained RL: When using memory-saving algorithms like GRPO, which eliminate the critic network by computing relative advantages inside sampled groups.

When to Avoid

• Subjective Tasks: Creative writing, summarization, or general dialogue where "correctness" cannot be programmatically validated.
• Weak Base Models: If the base model has zero probability of generating the correct answer initially, RLVR cannot explore effectively (the "cold start" problem). SFT is required first.
• Susceptibility to Spurious Rewards: Recent research shows some model families (e.g., Qwen) can exploit clipping biases to amplify pre-existing behaviors even when rewards are completely random/spurious.

History & Evolution

FEBRUARY 2024

DeepSeekMath & GRPO Introduction

DeepSeekMath 7B demonstrates that mathematical reasoning can be dramatically enhanced using Group Relative Policy Optimization (GRPO). By comparing a group of outputs against each other, GRPO eliminates the massive memory footprint of a separate critic model, paving the way for scalable RLVR.

JANUARY 2025

DeepSeek-R1: Pure RL Reasoning Emergence

DeepSeek-R1 demonstrates that pure RLVR can incentivize advanced reasoning patterns (self-reflection, verification, and dynamic strategy adaptation) without relying on extensive human-annotated demonstrations, achieving state-of-the-art performance on math and coding benchmarks.

APRIL – JUNE 2025

Critical Evaluations & Limits of RLVR

Subsequent papers critically analyze RLVR. Research shows that current algorithms do not necessarily elicit fundamentally new reasoning patterns beyond the base model's latent capabilities (bounded by the base model's $pass@k$ performance). Furthermore, studies reveal that models can exploit spurious rewards (even random ones) due to clipping biases in GRPO, highlighting model-dependent behaviors.

Implementation Sketch

grpo_verifiable_rewards.py Python / PyTorch

import torch
import torch.nn as nn

def compute_verifiable_reward(prompt, generation, ground_truth):
    """
    Deterministic verification function (e.g., symbolic math check or code execution)
    """
    extracted_answer = extract_answer(generation)
    if not extracted_answer:
        return 0.0
    return 1.0 if verify_equivalence(extracted_answer, ground_truth) else 0.0

def train_grpo_step(policy, ref_policy, prompts, ground_truths, group_size=5, beta=0.1):
    # 1. Sample a group of outputs for each prompt
    all_rewards = []
    all_log_probs = []
    
    for prompt, gt in zip(prompts, ground_truths):
        # Sample G outputs from current policy
        outputs = policy.generate(prompt, num_return_sequences=group_size)
        
        rewards = [compute_verifiable_reward(prompt, out, gt) for out in outputs]
        log_probs = policy.compute_log_probs(prompt, outputs)
        
        all_rewards.append(torch.tensor(rewards))
        all_log_probs.append(log_probs)
        
    # 2. Compute relative advantages within each group
    advantages = []
    for rewards in all_rewards:
        mean = rewards.mean()
        std = rewards.std() + 1e-8
        group_adv = (rewards - mean) / std
        advantages.append(group_adv)
        
    # 3. Calculate GRPO loss (simplified)
    loss = 0.0
    for i, (log_p, adv) in enumerate(zip(all_log_probs, advantages)):
        # Calculate policy ratios and KL divergence against reference policy
        ref_log_p = ref_policy.compute_log_probs(prompts[i], outputs)
        ratio = torch.exp(log_p - log_p.detach()) # r_i(theta)
        
        kl = log_p - ref_log_p
        
        # Clipped surrogate objective
        surr1 = ratio * adv
        surr2 = torch.clamp(ratio, 1.0 - 0.2, 1.0 + 0.2) * adv
        policy_loss = -torch.min(surr1, surr2).mean()
        
        loss += policy_loss + beta * kl.mean()
        
    return loss / len(prompts)

Common Variants

PPO + Verifiable Rewards

Standard Actor-Critic

Uses a separate neural network (Critic) to estimate state values. Highly stable but memory-intensive due to storing two massive models in GPU memory.

GRPO

Group-Relative (DeepSeek)

Eliminates the critic network entirely. Approximates the baseline value by averaging rewards across multiple outputs sampled from the same prompt.

Rejection Sampling (Best-of-N)

Offline SFT Pipeline

Generates $N$ candidates, filters out failed ones using the verifier, and fine-tunes the model on the successful trajectories via supervised learning.

Foundation & Related Corpus Papers

While the specific RLVR algorithms (like GRPO) are recent 2024–2025 developments, they build directly on foundational language model architectures and pre-training objectives in our corpus:

ryhchkpe82

Language Models are Few-Shot Learners

GPT-3 Foundations

Established the baseline zero-shot and few-shot reasoning capabilities in large language models that RLVR techniques subsequently optimize and unlock.

yudgctsnk7

Attention Is All You Need — Explaining the Transformer

Transformer Architecture

The fundamental sequence-to-sequence architecture that acts as the policy network ($\pi_\theta$) in all modern RLVR and reasoning frameworks.

Grounding References

[arXiv:2501.12948] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
[arXiv:2402.03300] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
[arXiv:2506.10947] Spurious Rewards: Rethinking Training Signals in RLVR
[arXiv:2504.13837] Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?