Soft Actor-Critic:
Off-Policy Maximum Entropy Deep RL with a Stochastic Actor
By Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, Sergey Levine • UC Berkeley • 2018
TL;DR
Soft Actor-Critic (SAC) stabilizes off-policy reinforcement learning by incorporating an entropy maximization term directly into the objective function. This encourages the agent to discover multiple near-optimal behaviors, dramatically improving sample efficiency and preventing premature policy collapse on complex continuous control tasks.
Why This Matters
Traditional deep reinforcement learning algorithms suffer from two main bottlenecks: high sample complexity (requiring millions of interactions with the environment) and extreme sensitivity to hyperparameters.
On-policy methods like PPO are stable but discard collected data immediately, making them highly sample-inefficient. Off-policy methods like DDPG reuse data from a replay buffer but are notoriously brittle, often failing to converge due to complex interactions between the deterministic actor and the high-dimensional Q-function approximator.
SAC bridges this gap by combining off-policy sample efficiency with a stochastic maximum entropy formulation that stabilizes learning.
The Big Idea
The core insight of Soft Actor-Critic is to change what the agent is trying to optimize. Instead of just maximizing the expected sum of rewards, the agent also tries to maximize the entropy of its actions.
In other words: succeed at the task while acting as randomly as possible.
The modified objective is defined as:
Where $\mathcal{H}(\pi(\cdot | s_t))$ is the entropy of the policy, and $\alpha$ is the temperature parameter that determines the relative importance of the entropy term against the reward.
How It Works
1. Soft Policy Iteration
The theoretical foundation of SAC relies on proving that alternating between Soft Policy Evaluation (calculating the soft Q-value under the current policy) and Soft Policy Improvement (updating the policy toward the exponential of the new soft Q-function) converges to the optimal policy.
Interactive Architecture Map
Hover over the components to see their mathematical role in SAC.
Click a component
Click any of the blocks in the diagram to inspect their mathematical formulation and role in the Soft Actor-Critic algorithm.
2. Action Bounding (The $\tanh$ squashing trick)
Because real-world physical systems have limited actuator limits, action outputs must be bounded (e.g., to $[-1, 1]$). SAC achieves this by passing an unbounded Gaussian action sample $u \sim \mathcal{N}(\mu, \sigma^2)$ through an invertible squashing function: $a = \tanh(u)$.
To compute the exact likelihood of the bounded action, SAC applies the change of variables formula:
Interactive Explorations
Entropy & Temperature ($\alpha$) Visualizer
See how changing the temperature parameter $\alpha$ balances exploration and exploitation over actions with different raw rewards.
Mathematical Result:
Action distribution: $P(A) \propto e^{R_A/\alpha}, P(B) \propto e^{R_B/\alpha}$.
Calculated Entropy: 0.00 nats.
Gaussian Squashing ($\tanh$) Visualizer
See how an unbounded action distribution is squashed into a bounded action space $[-1, 1]$. Notice how high variance causes actions to cluster near the boundaries.
Change of Variables Correction:
$\log \pi(a|s) = \log \mu(u|s) - \log(1 - \tanh^2(u))$
Exploration vs. Exploitation Simulator
Compare standard RL (greedy policy) against Maximum Entropy RL (SAC). Watch how standard RL gets stuck in the nearest local optimum, while SAC discovers both peaks due to its entropy incentive.
Empirical Results
SAC was evaluated on challenging continuous control benchmarks from OpenAI Gym and rllab, outperforming both on-policy (PPO) and off-policy (DDPG, TD3) baselines.
Performance Comparison (Average Return)
Limitations & Open Questions
- Sensitivity to Reward Scaling: SAC is highly sensitive to the scale of the reward function. The reward scale acts as the inverse of the entropy temperature $\alpha$. If the scale is too high, the policy becomes nearly deterministic; if too low, the policy becomes nearly uniform. (Note: Later iterations of SAC solved this via automated temperature tuning).
- Computational Overhead: Training two distinct Q-networks (Double Q-learning) and maintaining target networks increases the computational cost per gradient step.
- Continuous Action Space Constraint: The original formulation is strictly designed for continuous action spaces, requiring modifications to work on discrete actions.
Glossary
Entropy ($\mathcal{H}$)
A mathematical measure of randomness or uncertainty in a probability distribution. In RL, high policy entropy means the agent's actions are highly diverse and exploratory.
Off-Policy Learning
Algorithms that can learn from data collected by a different policy (e.g., historical transitions stored in a replay buffer), drastically improving sample efficiency.
Double Q-Learning
A technique that trains two independent Q-value estimators and uses the minimum of their predictions to prevent systematic overestimation bias during value updates.
Citation
@inproceedings{haarnoja2018soft,
title={Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor},
author={Haarnoja, Tuomas and Zhou, Aurick and Abbeel, Pieter and Levine, Sergey},
booktitle={International conference on machine learning},
pages={1861--1870},
year={2018},
organization={PMLR}
}