Visual Instruction Tuning
By Haotian Liu*, Chunyuan Li*, Qingyang Wu, Yong Jae Lee
TL;DR
LLaVA (Large Language and Vision Assistant) bridges the gap between vision and language by connecting a pre-trained visual encoder (CLIP) with a language model (Vicuna). By leveraging text-only GPT-4 to convert standard image-caption datasets into rich, multi-turn instruction-following data, LLaVA becomes one of the first end-to-end trained multimodal assistants capable of complex visual reasoning.
Why This Matters
Large Language Models (LLMs) have revolutionized human-computer interaction by serving as a universal interface. Through instruction tuning, models like ChatGPT can execute a wide range of tasks based on natural language prompts.
However, humans interact with the world multimodally. Traditional vision-language models were designed for highly specific, isolated tasks—like image captioning or object detection—with rigid interfaces. They lacked the flexibility to chat naturally about an image, reason through visual jokes, or solve step-by-step science puzzles. LLaVA addresses this by extending instruction tuning to the multimodal space, creating an assistant that can "see" and "reason" simultaneously.
The Big Idea
"How do we train a multimodal model to follow instructions when we don't have visual instruction-following datasets?"
The core breakthrough of LLaVA is its data generation pipeline. Since human-annotated multimodal instruction data is incredibly expensive and slow to collect, the authors used a clever workaround: they fed text-only representations of images (captions and bounding box coordinates) into GPT-4. By treating GPT-4 as a strong teacher, they generated 158,000 high-quality instruction-following samples spanning conversations, detailed descriptions, and complex reasoning. This synthetic data allowed them to train LLaVA end-to-end with minimal cost.
How It Works
1. GPT-Assisted Visual Instruction Data Generation
GPT-4 is text-only (at the time of the paper's development). To prompt it to generate visual conversations, the authors represented an image $X_v$ using two symbolic formats:
- Captions: Textual descriptions of the scene from multiple perspectives.
- Bounding Boxes: Coordinates $[x_{min}, y_{min}, x_{max}, y_{max}]$ locating objects in the scene.
By feeding these symbolic descriptions as context, GPT-4 was prompted to generate three distinct types of instruction-following data:
Symbolic Input to GPT-4
"A group of people standing outside of a black SUV with various luggage."
"People try to fit all of their luggage in an SUV in a public garage."
person: [0.681, 0.242, 0.774, 0.694]
backpack: [0.384, 0.696, 0.485, 0.914]
SUV: [0.100, 0.200, 0.850, 0.800]
GPT-4 Generated Output
2. Network Architecture
LLaVA connects a visual encoder with a large language model. It uses the pre-trained CLIP ViT-L/14 as its vision encoder and Vicuna as its language decoder.
To map the visual features into the word embedding space of the language model, LLaVA uses a simple, lightweight linear projection matrix $W$.
Click any component
Select a block in the diagram to inspect its exact role, dimensionality, and training behavior.
3. Two-Stage Training Procedure
Training a multimodal model from scratch can lead to instability and catastrophic forgetting. LLaVA bypasses this with a structured two-stage training paradigm:
Pre-training for Feature Alignment
Keeps both the visual encoder (CLIP) and LLM (Vicuna) frozen. Only the projection matrix $W$ is trained. This aligns visual features directly with word embeddings, essentially training a "visual tokenizer" for the LLM.
Dataset: 595K CC3M filtered subsetEnd-to-End Fine-Tuning
Keeps only the visual encoder frozen. Updates both the projection matrix $W$ and the LLM weights. This adapts the model to follow diverse multimodal instructions in a conversational format.
Dataset: 158K GPT-4 generated dataResults & Benchmarks
LLaVA was evaluated on two key benchmarks: LLaVA-Bench (COCO), which measures general conversational and reasoning capabilities, and ScienceQA, a multimodal science question-answering dataset.
ScienceQA Accuracy Comparison (%)
Interactive: Select categories to view performance breakdowns
Limitations & Open Questions
While LLaVA shows remarkable reasoning, the authors transparently note several limitations:
- The "Bag of Patches" Issue: LLaVA occasionally struggles with fine-grained spatial relationships, perceiving the image as a collection of features rather than a cohesive structure. For example, it might incorrectly identify strawberry-flavored yogurt in a fridge simply because both strawberries and yogurt are present elsewhere in the visual field.
- Resolution Limitations: Standard CLIP encoders process images at relatively low resolutions (e.g., $224 \times 224$ or $336 \times 336$ pixels), which can cause the model to miss small details, text, or complex textures.
- Hallucinations: Like all LLMs, LLaVA is prone to generating plausible-sounding but factually incorrect descriptions or reasoning steps not supported by the visual input.
Glossary
Instruction Tuning
A technique to train language models to follow user prompts or commands directly, aligning the model's outputs with human intent rather than just predicting the next token in a sequence.
Linear Projection Matrix (W)
A simple mathematical layer (a single linear neural network layer) used to map visual feature vectors from the vision encoder's dimensionality to match the word embedding space of the language model.
Vicuna
An open-source, instruction-tuned chatbot model built by fine-tuning Meta's LLaMA on user-shared conversations from ShareGPT.
Citation
@inproceedings{liu2023llava,
author = {Liu, Haotian and Li, Chunyuan and Wu, Qingyang and Lee, Yong Jae},
title = {Visual Instruction Tuning},
booktitle = {Neural Information Processing Systems (NeurIPS)},
year = {2023}
}