Reinforcement Learning Milestone

Dota 2 with Large Scale Deep Reinforcement Learning

By OpenAI • Published March 10, 2021 (arXiv)

TL;DR

OpenAI Five became the first AI system to defeat world champions in an esports game. By scaling deep reinforcement learning (PPO) to handle massive time horizons, partial observability, and high-dimensional spaces, the system learned to coordinate five distinct agents using self-play over a 10-month training process.

770 PFlops/s-days Compute 159M Parameters Model Surgery Self-Play PPO

Why This Matters

While board games like Chess and Go have been conquered by AI, they represent environments with perfect information and relatively short action horizons. Real-world challenges, however, are messy, continuous, and partially observed.

Dota 2 serves as a stepping stone to this real-world complexity. A single game lasts around 45 minutes (approx. 20,000 steps), compared to Chess (~80 moves) or Go (~150 moves). Players operate under a fog of war, meaning the state is only partially observed. Additionally, the action space is vast: at any given timestep, an agent must choose from 8,000 to 80,000 possible actions, managing complex resource allocation, real-time coordination, and long-term strategic plans.

The Big Idea

The core contribution of OpenAI Five is not a brand-new reinforcement learning algorithm, but rather proof that existing algorithms (specifically Proximal Policy Optimization, or PPO) can achieve superhuman performance when scaled to unprecedented levels.

To sustain a single training run over 10 months through changing game rules, code updates, and architecture expansions, OpenAI developed "Model Surgery"—a set of function-preserving network transformations that allowed training to continue seamlessly without restarting from scratch.

How It Works

1. The Model Architecture

Each of the five heroes on a team is controlled by a replica of the same policy network. The network takes in a semantic observation space of ~16,000 values (not pixels, but structured data representing units, items, and map state) and processes it into a single vector. This is then passed through a massive single-layer 4096-unit LSTM (accounting for 84% of the model's 159M parameters) to maintain memory of the game state.

Interactive Architecture Explorer

Hover over the blocks to inspect how OpenAI Five processes the game state into action outputs.

Select a block

Hover over any component of the neural network architecture on the left to see its parameters, inputs, and outputs.

2. Continual Transfer via Surgery

When training a model for 10 months, you cannot afford to restart training every time you tweak the network architecture, add a new game feature, or when the game developer releases a balance patch.

OpenAI solved this by performing Model Surgery. This technique maps parameters $\theta$ from an old model to a new model $\hat{\theta}$ such that the mathematical policy function remains identical:

$$\forall o \quad \hat{\pi}_{\hat{\theta}}(o) = \pi_{\theta}(o)$$

For example, when expanding a fully-connected layer's output from dimension $d_y$ to $\hat{d}_y$, the new weight matrix $\hat{W}_1$ is initialized with the old weights $W_1$ plus random values $R()$ for the new dimensions. Crucially, the downstream matrix $\hat{W}_2$ is padded with $0$s for those new dimensions, ensuring they initially contribute nothing to the output and preserve policy behavior.

Model Surgery Visualizer

Toggle between the states to see how the weight matrices are padded to preserve functions.

First Layer Weight Matrix ($\hat{W}_1$)

Second Layer Weight Matrix ($\hat{W}_2$)

Showing original model state before surgery.

3. Long-term Credit Assignment

In Dota 2, actions taken in the first minute of the game (such as purchasing a specific item) can dictate the outcome of a team fight 30 minutes later. To solve this, OpenAI Five uses a discount factor $\gamma$ tuned to assign credit over extremely long horizons.

The time horizon $H$ (in seconds) over which the agent integrates future rewards is defined as:

$$H = \frac{T}{1 - \gamma}$$

where $T = 0.133\text{s}$ is the real game time corresponding to each policy step (one action every 4 frames).

GAE Horizon & Reward Discounting Simulator

Discount Factor ($\gamma$) 0.9993

Calculated Horizon ($H$)

190s

In Game Minutes

3.17 mins

Reward Credit Decay Curve

The red dashed line indicates the calculated horizon $H$, where future rewards are discounted to $1/e \approx 36.8\%$ of their value.

Results & Training Milestones

The scaling laws of OpenAI Five produced an agent that rapidly surpassed human capabilities. Over the course of its 10-month training run, the policy continuously improved, as measured by TrueSkill.

TrueSkill Progression over Compute

Hover over the milestones to see OpenAI Five's skill milestones as compute scaled.

Hover over a green/red node on the chart

Explore the milestones achieved at different compute budgets.

During the public OpenAI Five Arena event, the system played 7,257 competitive games against the community, securing an astonishing 99.4% win rate.

Limitations & Open Questions

Despite its superhuman performance, OpenAI Five operated under specific constraints:

Restricted Hero Pool: Played with a subset of 17 heroes out of the full 117 available in Dota 2.
No Multi-Unit Control: Items or abilities allowing temporary control of multiple units (such as Illusion Rune or Necronomicon) were disabled to avoid technical complexity.
Sample Inefficiency: The model required 770 PFlops/s-days of compute. Humans learn to play Dota 2 with a tiny fraction of this data, indicating that current model architectures are highly sample-inefficient.

Glossary

PPO (Proximal Policy Optimization)

An on-policy reinforcement learning algorithm developed by OpenAI. It uses a clipped surrogate objective function to ensure that policy updates do not deviate too far from the previous policy, stabilizing training.

Model Surgery

A collection of offline parameter mapping tools that allow developers to alter a neural network's architecture (e.g., adding inputs, expanding layers) while preserving the exact policy outputs, preventing the need to retrain from scratch.

Staleness

The parameter version gap ($M - N$) between the model version $N$ used to generate rollout data and the model version $M$ currently being optimized. High staleness severely degrades optimization speed.

Citation

@article{openai2019dota,
  title={Dota 2 with Large Scale Deep Reinforcement Learning},
  author={OpenAI and Berner, Christopher and Brockman, Greg and Chan, Brooke and Cheung, Vicki and D{\k{e}}biak, Przemys{\l}aw and Dennison, Christy and Farhi, David and Fischer, Quirin and Hashme, Shariq and others},
  journal={arXiv preprint arXiv:1912.06680},
  year={2019}
}