Generative Adversarial Networks (GAN)

In One Sentence

GANs train a generator to synthesize realistic data from random noise by simultaneously training a discriminator to distinguish real training data from the generator's fakes, driving both networks to improve through competitive optimization.

Intuition

Imagine a game between a counterfeiter (the Generator) and a detective (the Discriminator).

The counterfeiter's goal is to print fake currency that looks indistinguishable from real money. The detective's goal is to inspect currency and accurately determine whether it is genuine or fake.

At first, the counterfeiter makes crude, obvious fakes. The detective easily catches them. However, as the detective points out the flaws, the counterfeiter learns and produces more sophisticated fakes. Simultaneously, to keep up, the detective must refine their own methods of detection. This adversarial loop continues until the counterfeiter's work is flawless, leaving the detective with nothing but a 50/50 guess to determine authenticity.

Key Advantage

Unlike traditional generative models (such as Variational Autoencoders), GANs do not require explicit probability density estimation, approximate inference networks, or Markov chain sampling. They learn to generate high-fidelity samples directly through backpropagation.

Formal Definition

Let \( p_{\text{data}} \) be the distribution over real data \( x \). The generator \( G(z; \theta_g) \) is a differentiable function represented by a multilayer perceptron that maps input noise variables \( z \sim p_z(z) \) to data space. The discriminator \( D(x; \theta_d) \) is another multilayer perceptron that outputs a single scalar representing the probability that \( x \) came from the training data rather than \( p_g \).

We train \( D \) to maximize the probability of assigning the correct label to both training examples and generated samples. We simultaneously train \( G \) to minimize \( \log(1 - D(G(z))) \). This formulation defines a minimax game with the value function \( V(D, G) \):

\min_G \max_D V(D, G) = \mathbb{E}_{x \sim p_{\text{data}}(x)}[\log D(x)] + \mathbb{E}_{z \sim p_z(z)}[\log (1 - D(G(z)))]

In the space of arbitrary functions \( G \) and \( D \), a unique global optimum exists where the generator's distribution \( p_g \) matches the data distribution \( p_{\text{data}} \) perfectly. At this equilibrium point:

p_g = p_{\text{data}} \quad \text{and} \quad D^*(x) = \frac{1}{2} \quad \forall x

Interactive Walkthrough

Visualizing the Minimax Convergence

Adjust the generator's training progress. Watch how the generator's distribution \( p_g \) aligns with the real data distribution \( p_{\text{data}} \), and how the optimal discriminator \( D^*(x) = \frac{p_{\text{data}}(x)}{p_{\text{data}}(x) + p_g(x)} \) flattens to a constant \( 0.5 \).

Generator Training Progress: 0%

Generator Mean (\(\mu_g\)): 2.0

Discriminator Decision: Highly Confident

Discriminator at Equilibrium: D(x) → 0.5

When to use / When not to

Use When...

You need high-fidelity, sharp, and visually realistic samples (e.g., image synthesis).
The underlying probability density function is too complex to model explicitly.
No explicit likelihood estimation is required for your downstream application.

Avoid When...

You require stable, guaranteed training convergence (GAN optimization is highly sensitive and prone to oscillation).
Mode Collapse is a critical failure mode (the generator learns to produce samples from only a few modes of the target distribution).
You need discrete data generation (e.g., text), as backpropagation through discrete choices is non-trivial.

History & Attribution

June 2014

Introduction of Generative Adversarial Networks

Goodfellow et al. (arXiv:1406.2661) propose the original GAN framework. They demonstrate that representing both the generator and discriminator as multilayer perceptrons allows training via backpropagation without Markov chains.

June 2016

Generative Adversarial Imitation Learning (GAIL)

Ho & Ermon (arXiv:1606.03476) draw an analogy between imitation learning and GANs, introducing a model-free framework to extract policies directly from expert demonstrations.

Pseudocode

gan_training_loop.py PyTorch-style

import torch

def train_gan(generator, discriminator, dataloader, num_epochs, latent_dim):
    # Optimizers
    opt_g = torch.optim.Adam(generator.parameters(), lr=0.0002, betas=(0.5, 0.999))
    opt_d = torch.optim.Adam(discriminator.parameters(), lr=0.0002, betas=(0.5, 0.999))
    
    loss_fn = torch.nn.BCELoss()

    for epoch in range(num_epochs):
        for real_data, _ in dataloader:
            batch_size = real_data.size(0)
            
            # --- Train Discriminator ---
            opt_d.zero_grad()
            
            # Real samples
            labels_real = torch.ones(batch_size, 1)
            out_real = discriminator(real_data)
            loss_real = loss_fn(out_real, labels_real)
            
            # Fake samples
            z = torch.randn(batch_size, latent_dim)
            fake_data = generator(z)
            labels_fake = torch.zeros(batch_size, 1)
            out_fake = discriminator(fake_data.detach())
            loss_fake = loss_fn(out_fake, labels_fake)
            
            loss_d = loss_real + loss_fake
            loss_d.backward()
            opt_d.step()
            
            # --- Train Generator ---
            opt_g.zero_grad()
            
            # Generator wants discriminator to output 1 (real) for fakes
            out_fake_g = discriminator(fake_data)
            loss_g = loss_fn(out_fake_g, labels_real)
            
            loss_g.backward()
            opt_g.step()

Common Variants

DCGAN

Deep Convolutional GAN. Replaces dense layers with spatial convolutions and transposed convolutions to stabilize image synthesis.

Wasserstein GAN (WGAN)

Uses Earth Mover's Distance instead of Jensen-Shannon divergence to provide smooth gradients everywhere, mitigating training instability.

GAIL

Generative Adversarial Imitation Learning. Adapts adversarial training to reinforcement learning to match agent trajectories to expert policies.

Representative Flash Papers

Primary Reference

Generative Adversarial Nets — Explained

We propose a new framework for estimating generative models via an adversarial process, in which we simultaneously train two models: a generative model G...

Alternative Paradigm

Auto-Encoding Variational Bayes — Explained

How can we perform efficient inference and learning in directed probabilistic models, in the presence of continuous latent variables...