CVPR 2024 / Under Review

Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction

Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, Liwei Wang Peking University & Bytedance Inc June 2024

"We present Visual Autoregressive modeling (VAR), a new generation paradigm that redefines the autoregressive learning on images as coarse-to-fine 'next-scale prediction', diverging from the standard raster-scan 'next-token prediction'."

GPT-style AR Power-Law Scaling 20x Faster than VQGAN Zero-Shot Editing

Why This Matters

Autoregressive (AR) models like GPT have completely dominated natural language processing due to their exceptional capability to scale up and perform zero-shot generalization. However, in computer vision, AR models have historically lagged behind diffusion models (like Stable Diffusion or DiT).

The core issue lies in how we represent images for sequence modeling. Standard vision AR models flatten a 2D image into a 1D sequence of tokens using a raster-scan order (left-to-right, top-to-bottom). This introduces three critical flaws:

Premise Violation: Flattening bidirectional 2D spatial features into a unidirectional sequence forces the model to learn artificial directional dependencies.
Structural Degradation: Raster scanning completely destroys spatial locality, separating pixels that are physically close to each other.
Extreme Inefficiency: Generating an image of resolution $n \times n$ pixel-by-pixel requires $O(n^2)$ sequential steps, resulting in a staggering computational complexity of $O(n^6)$.

The Big Idea

Instead of predicting the next token in a 1D sequence, why not predict the next scale of the image? Humans perceive and paint images from coarse structures to fine details.

VAR redefines image generation as a sequence of increasingly higher-resolution token maps:

$$p(r_1, r_2, \dots, r_K) = \prod_{k=1}^K p(r_k \mid r_1, r_2, \dots, r_{k-1})$$

Each $r_k$ represents a complete 2D token map at scale $k$ (e.g., $1\times1 \to 2\times2 \to 4\times4 \to \dots \to 16\times16$). Within each scale, all tokens are generated in parallel. This changes the game: the number of autoregressive steps drops from $O(n^2)$ to just $O(\log n)$ (typically only 10 steps), and the computational complexity drops to $O(n^4)$.

How It Works

1. Standard Raster-Scan vs. VAR Next-Scale

Observe the difference in generation strategy below. Standard AR must generate every token sequentially, whereas VAR generates entire spatial scales at once, preserving 2D structure.

Standard Raster-Scan AR Step: 0 / 64

Sequential 1D path destroys spatial relations.

Visual Autoregressive (VAR) Scale: 0 / 5

Parallel 2D scale expansion preserves spatial features.

2. Multi-Scale VQVAE (Residual Quantization)

To make next-scale prediction mathematically sound, VAR uses a custom Multi-Scale VQVAE. Instead of encoding the image independently at each resolution, it uses a residual design.

The encoder extracts feature map $f$. We quantize it at the lowest resolution first to get $r_1$, reconstruct it, and then quantize the *difference* (residual) at the next higher resolution $r_2$. This guarantees that higher-scale maps only encode fine details not present in the coarser scales.

Interactive Coarse-to-Fine Reconstruction

Drag the slider to see how the multi-scale tokenizer reconstructs the image from coarse to fine.

Selected Scale: Scale 1 (1x1)

Scale 1
(1x1) Scale 2
(2x2) Scale 3
(4x4) Scale 4
(8x8) Scale 5
(16x16)

1x1 grid - Global average color

3. Scaling Laws & Parameter Counts

One of the most exciting breakthroughs of VAR is that it exhibits clear power-law scaling laws, matching the behavior of Large Language Models. As we scale up the model's depth $d$, the parameter count grows as $N(d) = 18 d w^2 = 73728 d^3$ (with width $w = 64d$).

Interactive Scaling Law Calculator

Adjust the model depth to see how parameter count, predicted test loss, and error rates scale.

Model Depth ($d$)

16 Layers

Hidden Width ($w$): 1024

Heads ($h$): 16

Parameters

302M

VAR-d16

Formula: $73728 \times d^3$

Predicted Metrics

Predicted Test Loss ($L_{\text{last}}$) 2.14

Token Error Rate ($Err_{\text{last}}$) 93.1%

Empirical Results

VAR models achieve a historic milestone: for the first time, GPT-style autoregressive models surpass Diffusion Transformers (DiT) in visual quality, speed, and scaling efficiency on ImageNet benchmarks.

ImageNet 256×256 Generation Benchmark

Lower FID (Fréchet Inception Distance) indicates higher fidelity image generation.

VQGAN (Standard AR Baseline) FID: 18.65

DiT-XL/2 (Diffusion Transformer) FID: 2.27

L-DiT-7B (7 Billion Params Diffusion) FID: 2.28

VAR-d30 (Ours) FID: 1.92

VAR-d30-re (Ours + Rejection Sampling) FID: 1.73

20x Faster than VQGAN

45x Faster than DiT-XL/2

350.2 Inception Score (IS)

Limitations & Open Questions

While VAR achieves a massive breakthrough, the authors point out several important directions for future work:

Frozen Tokenizer: In this study, the VQVAE tokenizer architecture was kept simple and frozen. Transitioning to advanced tokenizers (like Finite Scalar Quantization - FSQ) could yield even better generation quality.
Text-to-Image Generation: The current model is primarily validated on class-conditional generation (ImageNet). Integrating text prompts via encoder-decoder structures remains an ongoing direction of research.
Video Generation: Video generation is not implemented in this work, though the authors suggest extending VAR to "3D next-scale prediction" using 3D pyramids to handle spatial-temporal features efficiently.

Glossary

Autoregressive (AR) Model

A model that predicts future values in a sequence based on past values. In NLP, this means predicting the next word given previous words. In standard vision, it means predicting the next pixel/token in a raster-scan sequence.

Fréchet Inception Distance (FID)

A metric used to assess the quality of images created by a generative model. It calculates the distance between feature vectors of real and generated images. Lower values mean the generated images look more realistic.

Quantization (VQVAE)

Vector Quantized Variational Autoencoder. It maps continuous image features onto a discrete set of "codebook" vectors, allowing continuous visual data to be represented as discrete tokens similar to words in a dictionary.

Citation

@article{tian2024visual,
  title={Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction},
  author={Tian, Keyu and Jiang, Yi and Yuan, Zehuan and Peng, Bingyue and Wang, Liwei},
  journal={arXiv preprint arXiv:2404.02905},
  year={2024}
}