Preprint 2025

The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models

By Ganqu Cui*, Yuchen Zhang*, Jiacheng Chen*, Lifan Yuan, Zhi Wang, and co-authors from Shanghai AI Laboratory, Tsinghua University, UIUC, Peking University, and CUHK.

TL;DR

Reinforcement Learning (RL) for reasoning LLMs suffers from rapid entropy collapse, locking the model into premature overconfidence and stalling performance. This paper mathematically proves that entropy decline is driven by the positive covariance between action probabilities and advantage changes. By introducing Clip-Cov and KL-Cov, the authors selectively suppress high-covariance update outliers, preventing collapse and unlocking massive scaling gains.

Entropy Collapse $R = -a\exp(H) + b$ Law Covariance Regularization Up to +15.0% on AIME

Why This Matters

In the quest to scale reasoning capabilities in Large Language Models (LLMs), Reinforcement Learning (RL) has emerged as the premier frontier. Unlike standard pre-training or supervised fine-tuning, RL allows models to discover novel, multi-step problem-solving paths through trial and error.

However, standard RL algorithms run into a brick wall very early in training. The model's policy entropy—a metric of its willingness to explore diverse options—collapses to near zero.

Once entropy collapses, the model becomes absolute in its choices. It stops exploring alternative reasoning paths, and its validation accuracy plateaus. This "exploration curse" prevents RL from effectively leveraging larger computational budgets.

The Big Idea

"The policy performance is traded from policy entropy, thus bottlenecked by its exhaustion."

The authors uncover a precise, predictable mathematical relationship governing this trade-off during RL training:

$R = -a \exp(H) + b$

Where $R$ is the validation performance, $H$ is the policy entropy, and $a, b$ are model-and-dataset-specific constants. This means that without intervention, performance gains are paid for directly in entropy consumption. Once entropy hits zero ($H \to 0$), performance hits a predictable ceiling of $-a + b$.

Interactive Widget 1: The Entropy-Performance Curve Simulator

Adjust parameters $a$ (entropy-to-performance conversion rate) and $b$ (theoretical offset) to see how model size affects the scaling trajectory. Slide the current Entropy ($H$) to visualize the "trade" towards the performance ceiling.

Parameter $a$ (Conversion Rate): 15.0

Parameter $b$ (Offset): 55.0

Current Entropy ($H$): 0.80

Calculated Metrics:

Current Accuracy ($R$): 21.6%

Max Ceiling ($H=0$): 40.0%

How It Works: The Entropy Dynamics

To stop entropy collapse, we must first understand what drives it. The authors analyzed the mathematical step-by-step changes in policy entropy for a softmax policy (which all LLMs use).

Theorem 1: Entropy Change under Policy Gradient

The step-wise difference in policy entropy between training step $k$ and $k+1$ is driven by the negative covariance between log-probabilities and logit changes:

$$\mathcal{H}(\pi_\theta^{k+1}|s) - \mathcal{H}(\pi_\theta^k|s) \approx -\eta \cdot \text{Cov}_{a \sim \pi_\theta^k} \left( \log \pi_\theta^k(a|s), \, \pi_\theta^k(a|s) \cdot A(s, a) \right)$$

Where $\eta$ is the learning rate, and $A(s, a)$ is the action advantage.

This derivation reveals a beautiful intuitive mechanism:

High-probability actions with high advantage reduce policy entropy (the model double-down on what it already knows works, leading to overconfidence).
Rare actions with high advantage increase policy entropy (the model is surprised by a hidden gem, prompting more exploration).

Unfortunately, early in training, LLMs are already well-calibrated. High-probability tokens reliably get positive advantages, meaning the covariance term stays persistently positive, dragging entropy down monotonically.

Interactive Widget 2: Covariance & Entropy Dynamics Simulator

Simulate 3 candidate actions. Adjust their current probabilities and advantages to see how they impact the covariance term and whether the next training step will collapse or boost the model's exploration.

Action A (Exploitation Target)

Probability: 0.70

Advantage: +2.0

Action B (The Surprise/Rare Path)

Probability: 0.10

Advantage: +1.5

Action C (Unpromising Path)

Probability: 0.20

Advantage: -1.0

Live Dynamics Calculation

Covariance Term:

+0.345

Entropy Trend:

Collapsing (-$\Delta H$)

Notice how increasing the probability of a high-advantage action (Action A) drives the covariance up, forcing entropy down. To increase entropy, you need to find rare actions (Action B) with high advantage!

Entropy Control by Covariance Regularization

Since standard entropy regularization (adding an entropy loss term) fails or is hyperparameter-sensitive in LLMs, the authors propose targeting the root cause: outlier tokens with massive positive covariance.

By identifying the top-ranking covariance tokens in each batch, they introduce two simple yet highly effective techniques:

Clip-Cov

Gradient Detachment

Randomly selects a small fraction of high-covariance tokens and detaches their gradients entirely. This prevents them from dominating the optimization step and collapsing the entropy.

KL-Cov

Targeted KL Penalty

Applies a localized Kullback-Leibler (KL) divergence penalty specifically to the top-$k$ highest covariance tokens, anchoring them to the reference policy and preserving exploratory potential.

Interactive Widget 3: Token-Wise Update Selection

See how standard GRPO compares to Clip-Cov and KL-Cov. Toggle the methods to watch how high-covariance outlier tokens (highlighted in red) are managed during gradient updates.

let

prime

(

)

return

true

;

Select a method above to visualize the update strategy.

Results & Performance Gains

The proposed methods completely reshape the entropy curve during training. Instead of collapsing to zero, policy entropy is maintained at a healthy level, allowing the model to continuously explore and improve.

The results are particularly striking when scaling to larger models, as they have more latent pre-trained behaviors waiting to be unlocked once the "exploration curse" is lifted.

Model	Method	AIME 24	AIME 25	MATH-500	Avg. Score
Qwen2.5-7B	GRPO (Baseline)	21.2%	9.6%	78.8%	38.6%
Qwen2.5-7B	KL-Cov (Ours)	22.6%	12.9%	80.8%	40.6% (+2.0)
Qwen2.5-32B	GRPO (Baseline)	21.8%	16.2%	84.2%	45.8%
Qwen2.5-32B	KL-Cov (Ours)	36.8%	30.8%	84.6%	52.2% (+6.4)

Limitations & Open Questions

Optimal Entropy Target: While the proposed methods effectively prevent collapse, it remains an open question whether there exists an mathematically optimal target entropy curve that perfectly balances exploration and training stability.
Hyperparameter Sensitivity: Policy entropy is highly sensitive to the precise thresholds of Clip-Cov and KL-Cov. Even though only $10^{-4}$ to $10^{-3}$ of the tokens are modified, tuning these parameters is crucial.
Verifiable Task Dependency: This study heavily relies on verifiable tasks (math and code) where the reward is binary and ground-truth-backed. Extending this to open-ended generation remains to be explored.

Glossary

Policy Entropy

A measure of the randomness or uncertainty in an agent's action selection. High entropy means the model spreads its probability over multiple candidate answers (exploration), while low entropy means it is highly confident in a single choice (exploitation).

Advantage Function

Represents how much better a specific action is compared to the average expected reward of the current state. A positive advantage means the action outperformed expectations.

Softmax Policy

A policy parametrization where action probabilities are computed by taking the exponential of the model's output logits and normalizing them across the action space. This is standard in modern autoregressive LLMs.

Citation

@misc{cui2025entropymechanismreinforcementlearning,
  title={The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models}, 
  author={Ganqu Cui and Yuchen Zhang and Jiacheng Chen and Lifan Yuan and Zhi Wang and Yuxin Zuo and Haozhan Li and Yuchen Fan and Huayu Chen and Weize Chen and Zhiyuan Liu and Hao Peng and Lei Bai and Wanli Ouyang and Yu Cheng and Bowen Zhou and Ning Ding},
  year={2025},
  eprint={2505.22617},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  url={https://arxiv.org/abs/2505.22617}, 
}