Alignment Reinforcement Learning

RLHF

Reinforcement Learning from Human Feedback

Aliases: reinforcement learning from human feedback

Intuition

Standard large language models are trained on next-token prediction over massive web corpora. While this teaches them grammar, facts, and reasoning, it does not make them helpful, honest, or harmless. A raw model will gladly generate toxic text, hallucinate plausible lies, or ignore instructions because it is merely imitating its training data.

RLHF bridges this gap by shifting the training objective from pure imitation to preference satisfaction. The process unfolds in three core steps:

1

Supervised Fine-Tuning

The base model is trained on high-quality demonstrations of instructions and desired answers (SFT step).

2

Reward Modeling

Humans rank multiple model-generated responses. A separate "Reward Model" is trained to output high scores for preferred answers and low scores for rejected ones.

3

Reinforcement Learning

The SFT model is optimized using an RL algorithm (typically PPO) to maximize the reward model's score, constrained by a penalty to prevent drifting too far from the original model.

Formal Definition

Let $x$ be a prompt, and let $y_w$ and $y_l$ be the preferred (winning) and dispreferred (losing) responses generated by the model.

1. Reward Model Loss

The reward model $r_\theta(x, y)$ is trained to predict human preferences using a binary cross-entropy loss based on the Bradley-Terry model:

$$\mathcal{L}(\theta) = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \left[ \log \sigma \left( r_\theta(x, y_w) - r_\theta(x, y_l) \right) \right]$$

where $\sigma(z) = \frac{1}{1 + e^{-z}}$ is the sigmoid function, and $\mathcal{D}$ is the dataset of human comparisons.

2. Reinforcement Learning Objective

Using the frozen reward model $r_\theta$, we fine-tune the SFT policy $\pi^{SFT}$ to obtain the RL policy $\pi_\phi^{RL}$. To prevent the policy from collapsing or generating gibberish that exploits the reward model ("reward hacking"), we add a Kullback-Leibler (KL) divergence penalty:

$$\text{obj}(\phi) = \mathbb{E}_{(x, y) \sim \mathcal{D}_{\pi_\phi^{RL}}} \left[ r_\theta(x, y) - \beta \mathbb{D}_{KL}\left(\pi_\phi^{RL}(y|x) \parallel \pi^{SFT}(y|x)\right) \right] + \gamma \mathbb{E}_{x \sim \mathcal{D}_{pretrain}} \left[ \log \pi_\phi^{RL}(x) \right]$$

where:

$\beta$ is the KL penalty coefficient controlling policy drift.
$\mathbb{D}_{KL}(P \parallel Q) = \sum P(x) \log \frac{P(x)}{Q(x)}$ is the KL divergence.
The final term (weighted by $\gamma$) is an optional pre-training gradient correction to prevent performance degradation on standard public NLP benchmarks.

When to Use & Trade-offs

Use When

Instruction Following: You need a model to accurately execute user prompts rather than just continuing the text.
Subjective Alignment: The target behavior (e.g., "helpfulness" or "safety") is hard to define with simple heuristic rules but easy for humans to rank.
Drastic Parameter Efficiency: You want a smaller model to perform on par with much larger models on human-evaluated tasks.

Avoid When

Objective Metrics Exist: If the task has a clear mathematical objective (e.g., code compilation, math execution), RL with programmatic verifiers is superior.
Resource Constraints: RL training (specifically PPO) is highly unstable, memory-intensive, and requires keeping multiple large models in GPU memory (SFT, RL, Critic, Reference).
Strict Determinism: RLHF can reduce the model's entropy, sometimes leading to repetitive or overly cautious "evasive" answers.

History & Key Milestones

2017

Proximal Policy Optimization (PPO)

Schulman et al. (1707.06347) introduce PPO, a stable and sample-efficient policy gradient method. PPO eventually becomes the standard reinforcement learning backbone used to optimize LLM policies against human preference reward models.

March 2022

InstructGPT: Aligning LMs with Human Feedback

Ouyang et al. (2203.02155) demonstrate that applying RLHF to GPT-3 yields massive alignment gains. Their 1.3B parameter InstructGPT model is preferred by human evaluators over the 175B base GPT-3, establishing RLHF as the standard paradigm for modern instruction-following LLMs.

December 2022

Constitutional AI (RLAIF)

Bai et al. (2212.08073) scale feedback collection by replacing human labelers with AI feedback (RLAIF). By guiding a model with a set of rules or "constitution," they train highly harmless and helpful assistants with minimal human labels.

Interactive Walkthrough: The KL Trade-off

In RLHF, the KL penalty ($\beta$) acts as a leash. If $\beta$ is too high, the model cannot learn to satisfy human preferences. If $\beta$ is too low, the model exploits flaws in the reward model—a phenomenon known as reward hacking—resulting in high reward scores but gibberish or incoherent output.

Simulating Policy Optimization

Adjust the KL penalty weight to see how it affects reward and final output quality.

KL Penalty ($\beta$) 0.15

Policy Drift (KL)

4.21

Model Reward Score

+2.84

Actual Human Utility

84%

Simulated Model Response

"To write a python script, you can use the built-in open function..."

Optimal Alignment

Pseudocode Implementation

A simplified PyTorch-like representation of the Reward Model training step and the subsequent PPO actor loss calculation with the KL penalty.

rlhf_optimization.py Python

import torch
import torch.nn.functional as F

# --- Step 1: Reward Model Loss ---
def compute_reward_loss(reward_model, prompt_ids, win_ids, lose_ids):
    # Get scalar rewards for both responses
    r_win = reward_model(prompt_ids, win_ids)    # Shape: (batch_size,)
    r_lose = reward_model(prompt_ids, lose_ids)  # Shape: (batch_size,)
    
    # Bradley-Terry preference loss
    loss = -torch.log(torch.sigmoid(r_win - r_lose)).mean()
    return loss

# --- Step 2: PPO Policy Update with KL Penalty ---
def compute_ppo_kl_loss(actor_logits, ref_logits, rewards, kl_coef=0.1):
    """
    actor_logits: Logits from the active RL policy being updated
    ref_logits: Logits from the frozen SFT reference model
    rewards: Scalar rewards from the frozen Reward Model
    """
    # Calculate log probabilities
    actor_log_probs = F.log_softmax(actor_logits, dim=-1)
    ref_log_probs = F.log_softmax(ref_logits, dim=-1)
    
    # Exact KL Divergence: D_KL(Actor || Reference)
    kl_div = torch.sum(
        torch.exp(actor_log_probs) * (actor_log_probs - ref_log_probs), 
        dim=-1
    )
    
    # Modify reward signal with KL penalty
    kl_penalized_reward = rewards - kl_coef * kl_div
    
    # Standard PPO policy gradient surrogate objective would follow:
    # loss = -surrogate_objective(actor_log_probs, kl_penalized_reward)
    return kl_penalized_reward.mean()

Common Variants

RLAIF (RL from AI Feedback)

Introduced in Constitutional AI (2212.08073), RLAIF replaces human preference labelers with an LLM evaluator guided by a set of principles (a "constitution"). It scales alignment dramatically while maintaining similar performance.

Direct Preference Optimization (DPO)

A modern mathematical shortcut that bypasses training a separate reward model and running unstable reinforcement learning. DPO optimizes the policy directly on preference pairs by mathematically reparameterizing the reward loss.

Representative Flash Papers

Explore how RLHF is applied to or relies on foundational architectures in our corpus:

ryhchkpe82

Language Models are Few-Shot Learners

Introduces GPT-3, the massive 175B parameter base model. This base model served as the raw starting point for InstructGPT, proving that scale alone is insufficient for alignment, but provides the ideal foundation for RLHF.