NeurIPS 2020

Denoising Diffusion Probabilistic Models

Jonathan Ho, Ajay Jain, Pieter Abbeel • UC Berkeley

"We present high quality image synthesis results using diffusion probabilistic models, a class of latent variable models inspired by considerations from nonequilibrium thermodynamics."

SOTA Image Quality Markov Chains Langevin Dynamics

Why This Matters

Before 2020, Generative Adversarial Networks (GANs) dominated high-fidelity image synthesis. However, GANs are notoriously difficult to train, prone to mode collapse, and lack a clear probabilistic framework. Other likelihood-based models, like VAEs and Flow models, offered stable training but often produced blurry samples or required highly constrained architectures.

This paper proved that Denoising Diffusion Probabilistic Models (DDPMs) could match or exceed GAN sample quality while maintaining a stable, likelihood-based training objective. By breaking down the daunting task of generating an image from scratch into hundreds of tiny, manageable denoising steps, DDPMs unlocked a new paradigm for generative AI.

The Big Idea

Imagine dropping food coloring into water. The ink starts concentrated (a clean image) and diffuses outward until it is completely mixed and uniform (pure noise). Reversing this physical process—gathering the dispersed ink back into its original shape—is incredibly difficult.

DDPMs solve this by training a neural network to perform this reversal step-by-step. Instead of trying to map pure noise directly to a clean image in one giant leap, the model learns to remove a tiny sliver of noise at each step, gradually coaxing structure out of chaos.

Interactive Playground

1 The Forward Process (Noising)

Equation (4) simulator

The forward process $q(x_t | x_0)$ adds Gaussian noise at each timestep $t$. Because of the nice properties of Gaussians, we don't have to step through iteratively; we can sample any step $t$ in closed form using: $$q(x_t|x_0) = \mathcal{N}(x_t; \sqrt{\bar{\alpha}_t}x_0, (1 - \bar{\alpha}_t)\mathbf{I})$$

Timestep ($t$): 500 / 1000

Beta schedule ($\beta_t$): 0.01

Cumulative variance ($\bar{\alpha}_t$): 0.50

Signal scale ($\sqrt{\bar{\alpha}_t}$): 0.71

Noise scale ($\sqrt{1 - \bar{\alpha}_t}$): 0.71

Simulated image $x_t$ at step $t$

2 The Reverse Process (Denoising)

Algorithm 2 Simulation

To generate a new image, we start with pure noise $x_T \sim \mathcal{N}(0, \mathbf{I})$ and run the reverse process. At each step, the neural network predicts the noise $\epsilon_\theta(x_t, t)$ that was added, allowing us to subtract it and step backward: $$x_{t-1} = \frac{1}{\sqrt{\alpha_t}} \left( x_t - \frac{\beta_t}{\sqrt{1 - \bar{\alpha}_t}} \epsilon_\theta(x_t, t) \right) + \sigma_t z$$

Sampling Progress

Current Step $t$: 1000 (Pure Noise)

Notice how structural layout (coarse details) emerges early in the reverse process (steps 1000 to 500), while fine details and textures are resolved at the very end (steps 200 to 0).

3 Simplified Objective ($L_{\text{simple}}$) vs. Variational Bound ($L$)

The key mathematical trick

The standard variational lower bound (VLB) heavily weights terms corresponding to small $t$. The authors discovered that dropping these weights and using a simplified MSE loss ($L_{\text{simple}}$) vastly improves sample quality: $$L_{\text{simple}}(\theta) = \mathbb{E}_{t, x_0, \epsilon} \left[ \| \epsilon - \epsilon_\theta(\sqrt{\bar{\alpha}_t}x_0 + \sqrt{1 - \bar{\alpha}_t}\epsilon, t) \|^2 \right]$$

Relative Weight of Loss Terms across Timesteps ($t$)

Small noise ($t \to 0$)

Large noise ($t \to T$)

Simplified Objective ($L_{\text{simple}}$) - Unweighted Standard Variational Bound ($L$) - Heavily downweights large $t$

Why does this work? By discarding the mathematical weighting, $L_{\text{simple}}$ down-weights loss terms corresponding to small $t$. This forces the network to focus on more difficult denoising tasks at larger $t$ steps, rather than spending its capacity on imperceptible high-frequency denoising.

How It Works

The U-Net Backbone

The neural network $\epsilon_\theta(x_t, t)$ is a U-Net with self-attention at the $16 \times 16$ resolution level. Because the exact same network is used for all timesteps $t$, the timestep is injected into the network using Transformer sinusoidal position embeddings, allowing the model to know exactly where it is in the diffusion trajectory.

Langevin Dynamics Connection

The authors showed that predicting the noise $\epsilon$ is mathematically equivalent to predicting the gradient of the data density (the score function). Thus, the reverse sampling process is essentially a form of annealed Langevin dynamics, carving paths through noise space towards high-density data regions.

Results & Metrics

DDPM achieved breakthrough results in unconditional image generation, particularly on the CIFAR10 dataset, beating contemporary GANs and likelihood-based models on sample quality metrics.

CIFAR10 FID

3.17

State-of-the-art at publication (unconditional)

Inception Score

9.46

Outperforming complex GAN baselines

LSUN Bedroom FID

4.90

Competitive with ProgressiveGAN

Limitations & Open Questions

Despite their incredible sample quality, DDPMs suffer from one major bottleneck: slow sampling speeds. Generating a single image requires running the U-Net backbone 1000 times sequentially. This is vastly slower than GANs or VAEs, which can generate images in a single forward pass.

Additionally, while DDPMs generate beautiful samples, their exact log-likelihoods are not as competitive as other likelihood-based models (such as autoregressive models or normalizing flows), because a large portion of the lossless codelength is consumed to describe imperceptible image details.

Glossary

Markov Chain

A stochastic model describing a sequence of possible events in which the probability of each event depends only on the state attained in the previous event. In DDPM, both the forward and reverse processes are modeled as Markov chains.

Langevin Dynamics

A mathematical framework for modeling the motion of molecular systems. In generative modeling, annealed Langevin dynamics is used to sample from a probability distribution by taking steps guided by the score function (gradients of the data density) plus added noise.

FID (Fréchet Inception Distance)

A metric used to assess the quality of images created by a generative model. It calculates the distance between feature vectors of real and generated images. Lower curves indicate higher quality and more realistic images.

Cite This Work

@inproceedings{ho2020denoising,
  title={Denoising Diffusion Probabilistic Models},
  author={Ho, Jonathan and Jain, Ajay and Abbeel, Pieter},
  booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
  year={2020}
}