μ
FlashPapers / MuZero
Original PDF
DeepMind • Nature 2020

Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model

Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, David Silver

TL;DR

MuZero introduces a paradigm shift in model-based reinforcement learning. Instead of trying to reconstruct entire environment screens or states, it learns an implicit model that predicts only the aspects critical for planning: the reward, the action-selection policy, and the value function. This allows it to match the superhuman performance of AlphaZero in chess, shogi, and Go, while simultaneously dominating visually complex Atari games—all without ever being told the rules of the environment.

Model-Based RL Value Equivalence MCTS Lookahead Latent Dynamics

Why This Matters

In reinforcement learning, agents are typically split into two camps: model-free and model-based.

Model-free agents learn strictly by trial and error, mapping states directly to actions. They excel in visually rich, complex environments like Atari games, but lack the ability to perform deep lookahead planning. Model-based agents, on the other hand, build a model of the world to plan ahead before taking actions. This is incredibly powerful in games like Chess or Go.

The Catch: Classic model-based planning requires knowing the rules of the game (a perfect simulator). In the real world, we rarely have a perfect simulator.

Historically, model-based methods that tried to *learn* the rules of visually rich environments struggled. They spent vast amounts of neural network capacity trying to reconstruct every pixel of the screen (such as background clouds, particle effects, or scoreboards)—details that are ultimately irrelevant for making the best decision.

The Big Idea

"Don't model the world. Model the utility of your decisions."

MuZero circumvents the pixel-reconstruction trap entirely. It does not try to predict what the next observation (pixels) will look like. Instead, it learns a model that projects observations into an abstract latent hidden state.

From this hidden state, it predicts only three critical values:

  • The Policy ($\mathbf{p}$): Which moves are best to play.
  • The Value ($v$): How likely are we to win from this state.
  • The Reward ($r$): What immediate points we will score.

By focusing only on these planning-relevant quantities, MuZero achieves value equivalence—its internal model of the game behaves identically to the real game for the purpose of planning, even if it looks nothing like it under the hood.

How It Works

1 The Tripartite Model Architecture

MuZero uses three distinct neural network functions to plan, act, and learn:

h_θ (Representation)

Transforms past observations $o_1 \dots o_t$ into an initial hidden state $s^0$.

s^0 = h_θ(o_1 \dots o_t)
g_θ (Dynamics)

Takes a hidden state $s^{k-1}$ and action $a^k$, predicting the next hidden state $s^k$ and reward $r^k$.

s^k, r^k = g_θ(s^{k-1}, a^k)
f_θ (Prediction)

Evaluates a hidden state $s^k$ to estimate the policy $\mathbf{p}^k$ and value $v^k$.

\mathbf{p}^k, v^k = f_θ(s^k)

Interactive Demo 1

Unrolling MuZero's Latent Model

o₁ ... o_t
h
s⁰ (Latent)
Vector s⁰
f
p⁰: [0.1, 0.7, 0.2]
v⁰: +0.45

Click Unroll Step to feed an action $a^k$ into the Dynamics function $g$ and generate the next latent state $s^k$.

2 Planning with Latent States (MCTS)

During action selection, MuZero performs a Monte Carlo Tree Search (MCTS). Crucially, this search takes place entirely within its learned latent space. It doesn't query the real environment. Starting from the current root state $s^0$, it simulates potential action sequences using the dynamics function $g_\theta$, and evaluates leaf nodes using the prediction function $f_\theta$.

Interactive Demo 2

MCTS Lookahead Tree Simulator

s⁰ N:0 Q:0.0 s¹_L N:0 Q:0.0 s¹_R N:0 Q:0.0

Click "Simulate Path" to run a lookahead simulation.

The agent selects actions by maximizing the Upper Confidence Bound (UCB) formula, updating visit counts ($N$) and mean values ($Q$) along the path.

3 The Power of the Latent Information Bottleneck

Why is MuZero so much more sample-efficient and robust than previous model-based RL systems? Traditional models try to reconstruct the pixel space. When the background contains visually complex but irrelevant details (e.g., flickering lights, moving clouds), pixel-reconstruction models waste huge capacity. MuZero's latent space acts as an information bottleneck, discarding everything that does not directly influence value, policy, or reward.

Interactive Demo 3

Pixel Reconstruction vs. Latent Dynamics

Pixel-Reconstructing Model
SCORE: 1240 LEVEL: 04
A

Spends network capacity trying to model the complex background noise. Reconstructions degrade as complexity rises.

MuZero Latent Model
Latent State s
Policy Head (p) 75% Confidence
Value Head (v) +0.92 Win Prob

Completely ignores background noise. Latent state remains clean, compact, and fully focused on planning utility.

Results & Benchmarks

MuZero was evaluated across two vastly different domains: classic board games (Chess, Shogi, Go) and 57 visually complex Atari 2600 games.

Classic Board Games

Superhuman

Evaluating MuZero against AlphaZero (which was supplied with game rules).

Go (Elo Rating) 5,400+

Slightly exceeded AlphaZero, despite using less computation per search node.

Chess & Shogi Matched AlphaZero

Atari 57 Games

State of the Art

Median human-normalized scores across all 57 games of the Arcade Learning Environment.

MuZero (Large Data) 2,041.1%
R2D2 (Previous SOTA - Model-free) 1,920.6%
MuZero Reanalyze (Sample Efficient, 200M frames) 731.1%

Smashed previous model-free bounds (Rainbow: 231.1%, LASER: 431%).

Limitations & Open Questions

While MuZero represents a massive step forward, the authors and subsequent researchers highlight several limitations:

  • Deterministic Latent Transitions: The dynamics function $g_\theta$ is fully deterministic. Extending MuZero to model highly stochastic environments (like weather patterns or multi-agent stock markets) remains an active area of research.
  • Computational Overhead: Although highly sample-efficient in terms of real environment steps, training MuZero requires substantial TPU/GPU compute power to perform the unrolled backpropagation-through-time and continuous MCTS simulations.
  • Compounding Latent Errors: In extremely long planning horizons, errors in the learned dynamics function $g_\theta$ can compound, leading the MCTS to explore unrealistic trajectories (though this is mitigated by value equivalence training).

Glossary

Value Equivalence

The principle that an abstract model does not need to reconstruct the true environmental states or observations, as long as planning over the abstract model yields the same optimal values and policies as planning in the real environment.

Monte Carlo Tree Search (MCTS)

A search algorithm used in decision-making processes, notably in gameplay. It builds a search tree of possible future states, evaluating them using random simulations or learned value networks.

Reanalyze

A sample-efficiency technique where MuZero re-runs MCTS searches on historical trajectories stored in the replay buffer using the latest, most advanced network weights to produce fresh, highly accurate training targets.

BibTeX Citation 
@article{schrittwieser2020mastering,
  title={Mastering Atari, Go, chess and shogi by planning with a learned model},
  author={Schrittwieser, Julian and Antonoglou, Ioannis and Hubert, Thomas and Simonyan, Karen and Sifre, Laurent and Schmitt, Simon and Guez, Arthur and Lockhart, Edward and Hassabis, Demis and Graepel, Thore and others},
  journal={Nature},
  volume={588},
  number={7839},
  pages={604--609},
  year={2020},
  publisher={Nature Publishing Group}
}
Made with Flash Papers — make your own ↗