Flamingo: a Visual Language Model for Few-Shot Learning
Jean-Baptiste Alayrac*, Jeff Donahue*, Pauline Luc*, Antoine Miech*, et al. — DeepMind
TL;DR: Flamingo is a state-of-the-art family of Visual Language Models (VLMs) that bridges frozen pretrained vision and language backbones. By using novel Perceiver Resamplers and gated cross-attention layers, Flamingo can perform open-ended multimodal tasks (VQA, captioning) from just a few visual-text examples in context, without any fine-tuning.
Why This Matters
Historically, adapting computer vision models to new tasks required thousands of high-quality annotated data points and resource-intensive, task-specific fine-tuning. While contrastive vision-language models like CLIP enabled zero-shot classification, they completely lacked the ability to generate language. This made them useless for open-ended tasks like image captioning, visual dialogue, or complex visual question answering.
In the text-only domain, Large Language Models (LLMs) like GPT-3 revolutionized adaptation through in-context few-shot learning—letting users define new tasks simply by writing a prompt with a few examples. Flamingo brings this exact superpower to the multimodal world. It allows a single model to adapt instantly to diverse, complex vision-language tasks using only a few interleaved images and sentences as a prompt.
The Big Idea
"Instead of training a massive multimodal network from scratch, why not reuse the best vision and language models we already have?"
Flamingo's core philosophy is model recycling. It freezes a state-of-the-art pretrained Vision Encoder (NFNet-F6) and a powerful pretrained Language Model (Chinchilla). It then inserts lightweight, trainable adaptation layers between them. This preserves the immense knowledge accumulated during expensive pretraining while establishing an expressive, stable pathway for visual features to guide text generation.
How It Works
Flamingo connects frozen backbones using two primary architectural innovations:
- The Perceiver Resampler: Ingests a variable number of spatio-temporal features from the vision encoder and maps them to a fixed set of 64 visual tokens. This keeps the cross-attention complexity constant regardless of input resolution or video length.
- Gated Cross-Attention (GATED XATTN-DENSE): Interleaved between frozen language model layers. It uses a learnable $tanh$ gate initialized to 0, ensuring that at the start of training, the model behaves exactly like the original language model, maintaining stability.
Interactive Architecture Explorer
Click different components of the Flamingo pipeline to see how data flows from pixels to words.
Click a component
Select any block in the pipeline diagram to explore its internal mechanics and design rationale.
Multi-Visual Input Support & Per-Image Masking
How does Flamingo handle prompts with multiple images interleaved with text without getting confused? The answer lies in their image-causal modeling. At any given text token, the model only cross-attends to the visual tokens of the image that appeared just before it in the sequence, rather than all previous images.
Mathematically, we define a function $\phi(l)$ which assigns to each text position $l$ the index of the last image appearing before that position:
This strict masking allows the model to scale seamlessly to any number of visual inputs at test time, even though it was only trained on sequences containing up to 5 images.
Interleaved Prompt & Masking Simulator
Hover over text tokens to see which preceding image they attend to under Flamingo's causal masking.
Hover over any text token above to see how Flamingo's cross-attention layers isolate and route visual information.
Results & Benchmarks
Despite using zero task-specific fine-tuning, the largest Flamingo model (Flamingo-80B) sets a new state of the art in few-shot learning across a wide array of benchmarks. Even more remarkably, on 6 out of 16 tasks, it outperforms models fine-tuned on thousands of times more task-specific data.
Performance Explorer
Compare Flamingo-80B few-shot performance against fine-tuned state-of-the-art baselines.
*On OKVQA, Flamingo-80B (32-shot) outperforms the fully supervised state-of-the-art despite using zero task-specific parameter updates.
Limitations & Open Questions
While Flamingo marks a massive leap forward, the authors transparently outline several critical limitations:
- Language Model Legacies: Because Flamingo builds on frozen LLMs, it inherits their weaknesses, including occasional hallucinations and ungrounded guesses.
- Poor Sequence Length Generalization: Transformers generalize poorly to test sequences significantly longer than their training window (2048 tokens). This makes high-shot prompting (e.g., 32 shots on long dialogue datasets like VisDial) perform worse than lower-shot prompts.
- Inference Compute Cost: Since it uses in-context learning, the compute cost scales linearly with the number of prompt examples (shots), making deployment expensive compared to compact fine-tuned models.
Glossary
Fine-Tuning
The process of taking a pre-trained model and training it further on a specific, narrower dataset to adapt its outputs to a particular task.
In-Context Few-Shot Learning
A paradigm where a model learns to perform a task simply by being shown a few input-output examples inside its prompt, without any gradient updates or parameter changes.
Vision Encoder
A neural network (often a CNN or Vision Transformer) that takes raw pixels as input and extracts high-level semantic features representing the visual scene.
Hallucination
A phenomenon where a generative model produces text that is fluent and grammatically correct, but factually incorrect or ungrounded in the provided input data.
Cite This Work
@inproceedings{alayrac2022flamingo,
title={Flamingo: a Visual Language Model for Few-Shot Learning},
author={Alayrac, Jean-Baptiste and Donahue, Jeff and Luc, Pauline and Miech, Antoine and Barr, Iain and Hasson, Yana and Lenc, Karel and Mensch, Arthur and Millican, Katie and Reynolds, Malcolm and others},
booktitle={Advances in Neural Information Processing Systems},
year={2022}
}