Hindsight Experience Replay
By Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, Wojciech Zaremba
"Dealing with sparse rewards is one of the biggest challenges in Reinforcement Learning. We present a novel technique called Hindsight Experience Replay which allows sample-efficient learning from rewards which are sparse and binary..."
Why This Matters
Imagine trying to learn how to play hockey. You shoot a puck, but it misses the net to the right. A standard reinforcement learning agent draws a simple, unhelpful conclusion from this episode: "That sequence of actions did not achieve the goal. Reward: 0 (or -1)." Because the reward is binary and sparse, the agent learns absolutely nothing from its failure.
To make standard RL work, researchers often rely on reward shaping—manually designing complex, continuous reward functions (e.g., rewarding the agent for moving its hand closer to the object, then the object closer to the target). However, reward shaping requires immense domain expertise, is highly prone to exploitation (where agents find unintended loopholes to maximize reward without solving the task), and limits the generalizability of RL systems.
The Big Idea
"If you can't reach the goal you wanted, pretend the place you ended up was your goal all along."
Humans possess a remarkable ability to learn from failure. When we miss a shot, we don't just learn "how not to hit the target"; we implicitly learn "how to hit the location where the puck actually landed."
Hindsight Experience Replay (HER) formalizes this intuition for artificial agents. During training, the agent stores its transitions in a replay buffer. Crucially, when replaying these transitions to update the policy, HER replaces the original goal with an alternative goal—such as the final state actually achieved during that episode. This transforms a completely failed attempt into a wildly successful one under a different context, allowing the agent to learn the physics of the environment even when it never hits the true target.
How It Works
HER is designed for Multi-Goal Reinforcement Learning. Instead of learning a policy $\pi(s)$ that maps states to actions, we train a Universal Value Function Approximator (UVFA), which learns a policy $\pi(s, g)$ conditioned on both the current state $s$ and a target goal $g \in \mathcal{G}$.
Let's explore this with the simplest multi-goal environment: the Bit-Flipping Environment. Here, the state is a sequence of $N$ bits. The action space consists of choosing an index $i \in [0, N-1]$ to flip. The reward is strictly binary: $0$ if the state matches the target goal, and $-1$ otherwise.
Interactive Demo 1: Bit-Flipping Environment
Sparse Reward VisualizationTry to match the Current State to the Target Goal. Standard RL struggles because random exploration almost never hits the exact goal when the number of bits increases.
Steps taken: 0 / 12
In standard RL, this entire trajectory yields only -1 rewards, offering no gradient for learning.
Goal Rewritten!
The replay buffer now stores a transition where the target goal equals the achieved final state. Under this virtual goal, the final step receives a reward of 0 (Success!). The agent now learns how to navigate the bit space!
Goal Sampling Strategies
How do we select which virtual goals to replay? The paper explores several strategies for sampling additional goals, controlled by a parameter $k$ (the ratio of hindsight transitions to actual transitions):
- Final: Replay with the final state achieved in the current episode.
- Future: Replay with $k$ random states encountered in the same episode *after* the transition being replayed.
- Episode: Replay with $k$ random states encountered anywhere in the same episode.
- Random: Replay with $k$ random states encountered across the entire training history.
Interactive Demo 2: Visualizing Goal Sampling
Select a transition along the trajectory to see which states are eligible to become virtual goals under different HER strategies.
Future Strategy
Replays the transition at $S_1$ using goals sampled from states observed after $S_1$ in the same episode (i.e., $S_2, S_3, S_4$). This ensures the goals are physically reachable from the current state, acting as an implicit curriculum.
Robotic Manipulation Environments
The authors validated HER on three challenging simulated robotic tasks using a 7-DOF Fetch arm in MuJoCo: Pushing, Sliding, and Pick-and-Place. In all tasks, the reward is binary: $0$ if the block is within a small tolerance of the target, and $-1$ otherwise.
Interactive Demo 3: Hindsight in Robotic Pushing
2D Physics SandboxClick "Simulate Push" to watch the robotic gripper attempt to push the block to the target. Notice how standard learning fails to capture the value of the trajectory when the block misses the target.
Even though the block missed the red target, HER creates a virtual transition where the target is placed exactly where the block stopped, teaching the model how to push to that location.
Headline Results
The empirical results demonstrate that HER is not just an optimization; it is an absolute requirement for solving complex manipulation tasks with sparse rewards.
Multi-Goal Success Rate
Success rate across training epochs (800 episodes/epoch)
Key Takeaway:
In the Pushing task, DDPG without HER completely fails to learn (0% success rate). DDPG with HER quickly achieves a near 100% success rate within 100 epochs.
Limitations & Open Questions
While HER is a massive leap forward for sparse-reward RL, it has several limitations:
- Assumption of Goal Reachability: HER assumes that any state achieved can be treated as a valid goal. In environments with irreversible dynamics (e.g., breaking a glass), this assumption breaks down.
- Off-Policy Constraint: HER is strictly compatible only with off-policy RL algorithms (like DQN or DDPG) because it relies on replaying transitions that do not match the current policy's target goal.
- Increased Memory Overhead: Storing multiple virtual transitions per actual step significantly increases the memory footprint of the replay buffer.
Glossary
Sparse Reward
A reward function that only provides feedback when a target is fully achieved (e.g., 0 for success, -1 otherwise), rather than providing continuous hints.
Universal Value Function Approximator (UVFA)
An extension of standard Q-learning where the value function is conditioned not only on the state $s$ and action $a$, but also on a goal $g$: $Q(s, a, g)$.
Off-Policy RL
Reinforcement learning algorithms that can learn from historical data generated by a different policy than the one currently being optimized.
@inproceedings{andrychowicz2017hindsight,
title={Hindsight experience replay},
author={Andrychowicz, Marcin and Wolski, Filip and Ray, Alex and Schneider, Jonas and Fong, Rachel and Welinder, Peter and McGrew, Bob and Tobin, Josh and Abbeel, Pieter and Zaremba, Wojciech},
booktitle={Advances in Neural Information Processing Systems},
pages={5048--5058},
year={2017}
}