Deep Residual Learning for Image Recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun • Microsoft Research • 2015
"Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously."
Max Depth Explored
1,202 Layers
ImageNet Top-5 Error
3.57%
Complexity (ResNet-152)
Lower than VGG-16
Why This Matters
In deep learning, depth is everything. As networks grow deeper, they naturally integrate low-, mid-, and high-level features in an end-to-end multi-layer fashion. However, in 2015, researchers hit a wall. Stacking more layers onto a network didn't just make optimization harder—it actually made performance worse.
This wasn't caused by overfitting, because the training error increased as well. Instead, it was a fundamental optimization bottleneck known as the degradation problem.
Interactive Degradation Simulator
See how increasing depth affects training error in standard "Plain" networks versus "Residual" networks. Drag the depth slider to observe the degradation problem.
The Big Idea
Instead of hoping stacked layers directly fit a desired underlying mapping $H(x)$, ResNet reformulates the problem. We let these layers fit a residual mapping:
The original mapping is recast into $F(x) + x$. This formulation is realized by feedforward neural networks with shortcut connections that perform identity mapping. This simple addition introduces zero extra parameters and zero computational complexity, yet completely alters the optimization landscape.
How It Works
The Residual Block
In a standard network, each layer learns a direct transformation of the input. If the optimal mapping is close to an identity function (i.e., doing nothing), the network must learn all weights from scratch to replicate that identity.
In a residual block, if identity is optimal, the optimizer can simply drive the weights of the nonlinear layers toward zero. The shortcut connection guarantees that information always flows forward, preconditioning the optimization problem.
Basic Block: Used in ResNet-18 and ResNet-34. Simple stack of two 3x3 convolutions.
The Gradient Highway Simulator
Visualize how gradients flow backward during training. Compare a Plain network with a Residual network to see why ResNet avoids the vanishing gradient problem.
In a Plain Network, gradients are multiplied by layer weights at every step backward. If weights are small, the gradient vanishes exponentially (represented by fading red arrows), leaving early layers with virtually no training signal.
Results & Benchmarks
ResNet swept the ILSVRC & COCO 2015 competitions, securing 1st place in ImageNet Classification, Detection, Localization, COCO Detection, and COCO Segmentation.
Top-1 and Top-5 error rates on the ImageNet validation set. Notice how ResNet error continues to drop as depth increases, while Plain network error rises.
Limitations & Open Questions
While ResNet revolutionized deep learning, the paper leaves open several areas for exploration:
- Overfitting on Aggressively Deep Nets: On small datasets like CIFAR-10, the 1202-layer network performed worse (7.93%) than the 110-layer network (6.43%) due to overfitting, indicating that massive depth requires proportional regularization or data scale.
- Diminishing Returns: Going from 101 to 152 layers on ImageNet yielded only minor accuracy gains compared to the jump from 34 to 50 layers.
- The Identity Assumption: The authors hypothesize that identity mapping is optimal, but later architectures (like DenseNet or ResNeXt) explore alternative shortcut combinations.
Glossary
Degradation Problem
A phenomenon where adding more layers to a deep network leads to saturated accuracy followed by rapid degradation of training performance, not caused by overfitting.
Shortcut Connection
Connections that skip one or more layers. In ResNet, these perform identity mapping, passing the input directly to the output of a block without adding parameters.
Bottleneck Block
A three-layer residual block design ($1\times1$, $3\times3$, $1\times1$ convolutions) where the $1\times1$ layers reduce and then restore dimensions, saving computational cost for deeper models.
Cite This Paper
@inproceedings{he2016deep,
title={Deep residual learning for image recognition},
author={He, Kaiming and Zhang, Xiangyu and Shaoqing, Ren and Sun, Jian},
booktitle={Proceedings of the IEEE conference on computer vision and pattern recognition},
pages={770--778},
year={2016}
}