Playing Atari with Deep Reinforcement Learning

Why This Matters

Before 2013, teaching machines to control complex environments directly from visual inputs was an elusive goal. Most successful Reinforcement Learning (RL) applications relied heavily on hand-crafted features combined with linear value functions. These systems were only as good as the features humans designed for them.

Deep learning had already revolutionized computer vision and speech recognition by extracting high-level features from raw data. However, applying deep learning to RL presented unique, fundamental challenges:

Sparse & Delayed Rewards

Unlike supervised learning's direct feedback, RL agents must learn from occasional, delayed scalar rewards.

Correlated Data

Consecutive game frames are highly correlated, violating the standard i.i.d. assumption of deep networks.

Non-Stationary Data

As the agent learns new behaviors, its actions change, shifting the distribution of states it encounters.

The Big Idea

The core contribution of Mnih et al. is the Deep Q-Network (DQN). It bridges the gap by training a deep convolutional neural network directly on raw video frames using a variant of Q-learning. To solve the correlation and non-stationarity issues, they introduced an Experience Replay mechanism that randomizes past transitions, effectively transforming the online RL problem into something resembling a stable, supervised learning task.

The Optimal Action-Value Function $Q^*(s,a)$

DQN approximates the expected future discounted return using the Bellman Equation as an iterative update:

$$Q^*(s, a) = \mathbb{E}_{s' \sim \mathcal{E}} \left[ r + \gamma \max_{a'} Q^*(s', a') \;\middle|\; s, a \right]$$

How It Works

1. State Preprocessing & Frame Stacking

Raw Atari frames are $210 \times 160$ RGB images. Operating directly on these is computationally heavy. Crucially, a single frame is also partially observable—you cannot determine the direction or speed of a ball from a single snapshot.

DQN solves this by converting frames to grayscale, downsampling to $110 \times 84$, cropping to an $84 \times 84$ playing area, and stacking the last 4 frames to form a state representation of shape $84 \times 84 \times 4$.

Interactive State Preprocessing Pipeline

Drag the slider to see how successive raw frames are processed and stacked to form the agent's input state.

1. Raw Frame (210x160 RGB)

SCORE 0042

2. Cropped & Gray (84x84)

3. Stacked State (84x84x4)

Frame $t-3$ Frame $t-2$ Frame $t-1$ Current Frame $t$

2. The DQN Architecture

Instead of feeding both the state and an action to the network to get a single Q-value, DQN uses an architecture that takes only the state representation as input and outputs a separate Q-value for each possible action. This allows the agent to compute the optimal action with a single forward pass.

Interactive Network Architecture

Hover over the layers of the network to inspect their specifications and spatial dimensions.

Input Layer

An $84 \times 84 \times 4$ image tensor produced by preprocessing the 4 most recent frames of the emulator history.

3. Experience Replay

To break temporal correlations and prevent divergence, DQN stores transitions $e_t = (s_t, a_t, r_t, s_{t+1})$ in a circular replay buffer $\mathcal{D}$. During training, minibatches of experience are sampled uniformly at random from this pool to update the network weights.

Replay Buffer Simulator

Click "Generate Transition" to add an agent experience to the buffer. Click "Sample Minibatch" to see how random sampling breaks temporal correlation.

Circular Buffer $\mathcal{D}$ (Max Capacity: 8)

Results & Evaluation

DQN was evaluated on seven Atari 2600 games. It outperformed all previous reinforcement learning techniques (such as Sarsa and Contingency) on six of the seven games, and even surpassed a human expert player on three of them.

DQN Performance vs. Baselines

Limitations & Open Questions

While DQN was a massive breakthrough, the authors highlighted several areas that remained challenging:

Long Time Horizons: Games like Q*bert, Seaquest, and Space Invaders require strategies that extend over long temporal scales. DQN struggled to plan far into the future due to the simple discount factor $\gamma$.
Reward Clipping: To stabilize gradients, all positive rewards were clipped to $+1$ and negative rewards to $-1$. This prevents the network from distinguishing between a small reward and a massive jackpot, potentially leading to suboptimal strategy decisions.
Uniform Sampling: The experience replay memory samples transitions uniformly. A more sophisticated approach would prioritize transitions from which the agent can learn the most (later conceptualized as *Prioritized Experience Replay*).

Glossary

Reinforcement Learning (RL)

An area of machine learning concerned with how intelligent agents ought to take actions in an environment to maximize the notion of cumulative reward.

Experience Replay

A technique where an agent's experiences are saved in a replay buffer. Minibatches are randomly sampled from this buffer to train the network, breaking temporal correlations in consecutive frames.

Partially Observable

An environment state is partially observable if the agent cannot perceive the complete state of the environment from its current sensor readings alone (e.g., needing velocity of a ball, not just its current position).

Citation

@article{mnih2013playing,
  title={Playing atari with deep reinforcement learning},
  author={Mnih, Volodymyr and Kavukcuoglu, Koray and Silver, David and Graves, Alex and Antonoglou, Ioannis and Wierstra, Daan and Riedmiller, Martin},
  journal={arXiv preprint arXiv:1312.5602},
  year={2013}
}