High-Resolution Image Synthesis with Latent Diffusion Models

1. Why This Matters

Generative models like GANs and autoregressive transformers have achieved incredible visual fidelity. However, their computational demands are staggering. Pixel-space Diffusion Models (DMs), while offering robust mode coverage and stable training, typically operate directly on raw RGB pixels.

This means training must spend massive representation capacity modeling imperceptible, high-frequency details (like individual strands of hair or complex textures) before learning the actual semantic structure of the scene. Consequently, training these models often consumes hundreds of GPU days, locking out researchers without massive industrial compute budgets.

2. The Big Idea

The core breakthrough of Latent Diffusion Models is the explicit separation of the image generation process into two distinct phases:

By transitioning from pixel space to latent space, training and inference become significantly more efficient, democratizing high-resolution synthesis.

3. How It Works

3.1 Perceptual Compression

The first stage trains an autoencoder on a massive collection of images. The encoder $\mathcal{E}$ maps an image $x \in \mathbb{R}^{H \times W \times 3}$ to a latent representation $z = \mathcal{E}(x) \in \mathbb{R}^{h \times w \times c}$, downsampling the spatial dimension by a factor $f = H/h = W/w$.

To prevent highly high-variance latent spaces, the authors experiment with two regularization techniques:

• KL-reg: Imposes a mild Kullback-Leibler penalty toward a standard normal distribution, similar to a standard VAE.
• VQ-reg: Integrates a vector quantization layer within the decoder, acting as a VQGAN but mapping the quantization directly into the decoder.

3.2 Latent Diffusion Models

Once the latent space is established, a time-conditional U-Net $\epsilon_\theta$ is trained to reverse a forward diffusion process. The objective function simplifies to predicting the noise added to the latent representation:

3.3 Conditioning via Cross-Attention

To turn diffusion models into highly flexible generators driven by multi-modal inputs (like text prompts, semantic layout maps, or bounding boxes), the authors augment the U-Net backbone with a cross-attention mechanism.

The input conditioning $y$ (e.g., text) is mapped via a domain-specific encoder $\tau_\theta(y)$ to an intermediate representation. This is then projected into the U-Net layers using:

4. Results & Benchmarks

Latent Diffusion Models achieve state-of-the-art results across various synthesis tasks while requiring significantly fewer parameters and training resources than pixel-space baselines.

5. Limitations & Societal Impact

Limitations

While LDMs significantly reduce computational requirements, their sequential sampling process during inference remains slower than single-pass generative models like GANs. Additionally, when high precision is required, the lossy nature of the autoencoder's reconstruction can become a bottleneck for tasks requiring pixel-perfect accuracy.

Societal Impact

By reducing the cost of training and deploying powerful generative models, this work democratizes access to advanced AI media tools. However, it also lowers the barrier to creating highly realistic manipulated imagery, raising concerns about misinformation, deepfakes, and intellectual property rights.

6. Glossary

7. Citation

@inproceedings{rombach2022high,
  title={High-resolution image synthesis with latent diffusion models},
  author={Rombach, Robin and Blattmann, Andreas and Lorenz, Dominik and Esser, Patrick and Ommer, Bj{\"o}rn},
  booktitle={Proceedings of the IEEE/CVF conference on computer vision and pattern recognition},
  pages={10684--10695},
  year={2022}
}