Neural Ordinary Differential Equations
Ricky T. Q. Chen*, Yulia Rubanova*, Jesse Bettencourt*, David Duvenaud
University of Toronto, Vector Institute
"Instead of specifying a discrete sequence of hidden layers, we parameterize the derivative of the hidden state using a neural network. The output of the network is computed using a black-box differential equation solver."
Why This Matters
Modern deep learning is fundamentally discrete. We stack layer upon layer—ResNet blocks, Transformer layers, attention heads—hoping that discrete steps will approximate the complex, continuous transformations of the real world. However, this discrete approach has severe limitations:
- Exploding Memory Costs: Backpropagating through deep networks requires storing all intermediate hidden activations. If you have 100 layers, you need 100 times the memory of a single layer.
- Rigid Computation: A standard network spends the same amount of compute on an easy input as it does on a hard one.
- Irregular Time-Series: Real-world data (like medical records or financial ticks) doesn't arrive at neat, discrete intervals. Traditional RNNs struggle to model continuous-time dynamics naturally.
The Big Idea
What if we took the limit of a Residual Network (ResNet) as the step size approaches zero?
A standard ResNet update is written as: $$h_{t+1} = h_t + f(h_t, \theta_t)$$ If we make the steps infinitely small, this discrete update formula transforms into an Ordinary Differential Equation (ODE): $$\frac{dh(t)}{dt} = f(h(t), t, \theta)$$ Instead of passing a vector through discrete layers, we define a continuous vector field using a neural network \(f\), and let a standard black-box ODE Solver (like Euler's method or Dormand-Prince) simulate the trajectory of our input state from start time \(t_0\) to end time \(t_1\).
How It Works
1. Discrete ResNet vs. Continuous Neural ODE
In a ResNet, the state jumps abruptly at each layer. In a Neural ODE, the state flows smoothly along a vector field defined by the network. Use the interactive widget below to compare how a discrete ResNet and a continuous Neural ODE transform an input point.
Interactive Demo 1
ResNet vs. Neural ODE Trajectory
2. Constant Memory Backpropagation via Adjoint
To train a Neural ODE, we need gradients of a loss function \(L\) with respect to the network parameters \(\theta\). Standard backpropagation requires storing all intermediate states of the forward pass in memory, which scales with depth: \(\mathcal{O}(L)\).
Instead, Neural ODEs use the Adjoint Sensitivity Method. We define the adjoint state as: $$a(t) = \frac{\partial L}{\partial z(t)}$$ Its dynamics are governed by another ODE, which is the instantaneous analog of the chain rule: $$\frac{da(t)}{dt} = -a(t)^T \frac{\partial f(z(t), t, \theta)}{\partial z}$$ We can solve this ODE backwards in time alongside the original state reconstruction, yielding gradients with constant \(\mathcal{O}(1)\) memory cost!
Interactive Demo 2
Memory Footprint: Standard Backprop vs. Adjoint Method
Standard backpropagation must cache all intermediate activations during the forward pass to compute gradients later.
3. Continuous Normalizing Flows (CNF)
Normalizing flows transform simple probability distributions (like standard Gaussians) into complex ones. In discrete flows, calculating the change in probability requires computing the determinant of the Jacobian matrix, which scales cubically: \(\mathcal{O}(D^3)\).
In a Continuous Normalizing Flow, the change in log probability follows an instantaneous change-of-variables formula: $$\frac{\partial \log p(z(t))}{\partial t} = -\text{tr}\left( \frac{\partial f(z(t), t, \theta)}{\partial z} \right)$$ Because the trace is a linear operation, we can sum the dimensions cheaply. This allows us to use wider hidden layers with a computational cost that scales only linearly with dimension!
Interactive Demo 3
Continuous Flow Particle Simulation
Key Results
The paper evaluated Neural ODEs across supervised learning, normalizing flows, and irregular time-series forecasting. Here are the standout comparisons:
MNIST Classification
ODE-Nets achieve the same accuracy (~0.42% error) as standard ResNets but with constant memory cost \(\mathcal{O}(1)\) during training, compared to ResNet's linear \(\mathcal{O}(L)\) cost.
Irregularly Sampled Data
When tested on irregular spiral data, the Latent ODE achieved a predictive RMSE of 0.1346, dramatically outperforming standard RNN baselines (RMSE of 0.1813).
Limitations & Open Questions
While Neural ODEs represent a beautiful theoretical leap, they are not a silver bullet. The paper points out several challenges:
- Increasing Computation Cost: As training progresses, the complexity of the learned vector field increases, requiring the adaptive solver to take more steps. This can slow down training significantly.
- Minibatching Overhead: Standard neural networks batch inputs easily. For Neural ODEs, batching means solving a combined system of ODEs, where the solver must adapt to the slowest/most complex sample in the batch.
- Numerical Stability: Reconstructing trajectories backwards in time can sometimes suffer from numerical divergence if the dynamics are highly chaotic or non-reversible.
Glossary
Adjoint Sensitivity Method
A mathematical technique used to compute gradients of a function defined by the solution of an ODE. It avoids backpropagating through the internal solver steps by solving a secondary, augmented ODE backwards in time.
Continuous Normalizing Flow (CNF)
A generative model that transforms simple probability distributions into complex ones using continuous-time trajectories, replacing the expensive Jacobian determinant computation of discrete flows with a cheap trace operation.
Adaptive Step-Size Solver
An ODE integration algorithm that dynamically adjusts its step size (\(dt\)) to keep numerical error below a user-specified tolerance limit, executing fewer steps where the dynamics are simple and more steps where they are complex.
Citation
@inproceedings{chen2018neuralode,
title={Neural Ordinary Differential Equations},
author={Chen, Ricky T. Q. and Rubanova, Yulia and Bettencourt, Jesse and Duvenaud, David},
booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
year={2018}
}