ICML 2017 Oral

Curiosity-driven Exploration by Self-supervised Prediction

Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, Trevor Darrell — UC Berkeley

TL;DR

The paper introduces the Intrinsic Curiosity Module (ICM), which generates a curiosity reward based on how hard it is for the agent to predict the consequences of its actions. By learning a feature space that only encodes factors the agent can control, ICM avoids the "noisy TV" trap and enables exploration with sparse or zero extrinsic rewards.

💡 Self-Supervised Learning 📺 Noisy TV Immunity 🎮 Zero-Reward Play

Why This Matters

Standard reinforcement learning (RL) relies heavily on frequent feedback. If you are training an agent to navigate a complex maze, it might only get a reward ($+1$) when it reaches the exit. In large environments, a random-walking agent will almost never stumble upon this exit by chance. This is the sparse reward problem.

To solve this, researchers use intrinsic rewards like curiosity. However, early curiosity models calculated surprise based on raw pixel predictions. This led to the infamous "Noisy TV" problem: if an agent encounters a television screen displaying static noise, it will stand in front of it forever, mesmerized by the completely unpredictable pixel changes, failing to explore the rest of the world.

The Big Idea

"Only predict what you can affect, or what can affect you."

The authors propose to map raw observations into a lower-dimensional feature space $\phi(s)$ using a self-supervised proxy task. This feature space is trained specifically to predict the agent's own actions (an inverse dynamics model). Because environmental noise (like leaves blowing in the wind or static on a TV) cannot be influenced by the agent's actions, the feature encoder learns to completely ignore it.

How It Works

The Intrinsic Curiosity Module (ICM) consists of two main neural network sub-modules working together:

Interactive Architecture Explorer

Hover blocks to inspect

Hover over any component of the ICM architecture above

Discover how raw state observations flow through feature encoders, inverse models, and forward dynamics models to construct a robust curiosity signal.

1. The Inverse Dynamics Model

To prevent the agent from focusing on background noise, we first learn a feature map $\phi(s)$ that filters out irrelevant visual signals. We train an inverse model $g$ to predict the action $a_t$ taken by the agent to transition from $s_t$ to $s_{t+1}$:

$\hat{a}_t = g(s_t, s_{t+1}; \theta_I)$

Because the network is optimized solely to predict the agent's actions, the learned feature space $\phi(s)$ has no incentive to represent environmental elements that the agent cannot control.

2. The Forward Dynamics Model

In parallel, we train a forward model $f$ to predict the next state's *feature representation* $\phi(s_{t+1})$ given the current feature representation and action:

$\hat{\phi}(s_{t+1}) = f(\phi(s_t), a_t; \theta_F)$

The curiosity reward $r^i_t$ is defined as the mean squared error of this prediction:

$r^i_t = \frac{\eta}{2} \|\hat{\phi}(s_{t+1}) - \phi(s_{t+1})\|_2^2$

Interactive Demo: The Noisy TV Trap

See the "Noisy TV" problem in action. Move the agent around. Notice how the pixel-based curiosity gets trapped by the flickering TV, while ICM curiosity remains unaffected and encourages exploration.

Agent Noisy TV

Raw Pixel Space

Includes TV noise!

ICM Feature Space $\phi(s)$

Ignored

φ(A)

Clean feature representation

Pixel-Space Prediction Error (Curiosity Reward)

0.00

High prediction error keeps agent trapped here.

ICM Feature-Space Prediction Error (Curiosity Reward)

0.00

Spikes only on actual movement; ignores TV.

Experimental Results

The authors evaluated ICM across several challenging environments, including VizDoom and Super Mario Bros.

VizDoom Navigation (Sparse Reward)

In the dense setting, the agent is spawned close to the goal. Standard A3C can learn, but ICM explores much faster.

ICM + A3C (Proposed) 100% Success

ICM-Pixels + A3C (Pixel-based baseline) 100% Success

Vanilla A3C (No Curiosity) 70% Success

In the Very Sparse setting, the agent is spawned in a separate room 350 steps away from the goal. Standard A3C completely fails to find the goal (0% success), whereas the ICM-based agent achieves 100% success at convergence.

Even in the No Reward Setting (where no goal is specified at all), the agent learns to navigate corridors, open doors, and explore rooms simply to satisfy its curiosity.

Limitations & Open Questions

The Curiosity Blockade (Boredom): If an agent encounters an obstacle that is too complex to solve (like a deep pit in Mario Level-1 requiring a precise 15-key sequence), it receives no gradient information. It gets stuck, "boredom" sets in, and exploration halts.
Generalization to Night Levels: An agent trained on Level-1 of Mario (daytime) struggles when deployed "as-is" on Level-2 (nighttime) due to global visual changes, requiring fine-tuning to update its feature encoder.

Glossary

Intrinsic Motivation

A drive to perform behavior for its own sake (e.g., curiosity, exploration) rather than for external rewards (e.g., game points, food).

Inverse Dynamics Model

A model trained to predict the action taken by the agent to transition from the current state to the next state.

Noisy TV Problem

A failure mode in curiosity-driven learning where an agent gets trapped by highly unpredictable but completely irrelevant environmental noise (like static on a TV screen).

Cite this Paper

@inproceedings{pathakICML17curiosity,
  Author    = {Pathak, Deepak and Agrawal, Pulkit and
               Efros, Alexei A. and Darrell, Trevor},
  Title     = {Curiosity-driven Exploration by Self-supervised Prediction},
  Booktitle = {International Conference on Machine Learning (ICML)},
  Year      = {2017}
}