Why This Matters
While large language models have unified natural language processing via zero-shot task generalization, computer vision has historically remained fragmented. Most visual segmentation models are trained on narrow datasets for specific, predefined classes (like "car" or "person").
To build a true foundation model for computer vision, we need a system that can segment any object in any image without task-specific fine-tuning. This requires a unified formulation of the segmentation task, a flexible model architecture, and a dataset of unprecedented scale.
The Big Idea
The Segment Anything project introduces three unified pillars:
- The Promptable Task: Predict a valid segmentation mask given any prompt (points, bounding boxes, text, or freeform masks).
- The Segment Anything Model (SAM): A decoupled architecture featuring a heavy image encoder paired with a lightweight, real-time prompt encoder and mask decoder.
- The Data Engine & SA-1B: A model-in-the-loop annotation pipeline that generated over 1.1 billion high-quality masks across 11 million licensed images.
How It Works
Explore the core architectural components of the Segment Anything Model (SAM).
1. Decoupled Architecture
To support real-time interactive use in a web browser, SAM splits computation. A heavy image encoder (based on a pre-trained Vision Transformer, $ViT-H$) processes the image once to produce a $64 \times 64$ spatial embedding.
Lightweight prompt encoders map interactive inputs (points, boxes, or text) into continuous vector tokens, which are fused with the image embedding in a fast Transformer-based mask decoder running in just $\sim 50\text{ms}$.
Interactive Architecture Map
Hover over any component of SAM to inspect its design, compute budget, and execution latency.
Select a component
Hover over the diagram blocks to view technical details.
2. Resolving Ambiguity
A single point prompt can be highly ambiguous. For example, clicking on a shirt button could mean "segment the button", "segment the shirt", or "segment the person wearing the shirt".
To solve this, SAM is designed to predict three nested masks simultaneously (corresponding to subpart, part, and whole levels) along with a predicted IoU score representing the confidence of each mask.
Ambiguity Resolution Simulator
Click on the target point to see the three distinct hierarchical masks SAM generates to resolve ambiguity.
By generating multiple valid outputs, SAM avoids producing an unhelpful, blurry average of possible shapes.
3. Interactive Prompting Playground
SAM supports iterative refinement. Adding a foreground point expands the mask, while background points exclude specific regions.
Prompt Simulator
Click inside the canvas to add foreground (green) or background (red) points to guide the segmentation mask.
Key Results
SAM was evaluated across 23 diverse datasets. The model demonstrates remarkable zero-shot generalization capabilities, often performing on par with or superior to fully supervised models.
Zero-Shot Transfer Performance (mIoU)
*Comparison based on the mean IoU across diverse out-of-domain evaluation suites under standard interactive point prompt settings.
Limitations & Open Questions
While SAM marks a significant step forward, several limitations remain open for future work:
- Fine Structures: SAM can occasionally miss highly intricate or thin structures, sometimes hallucinating disconnected components.
- Semantic Understanding: While SAM segments objects perfectly, it does not natively output semantic category labels (e.g., "cat", "chair") without external classification heads.
- Real-time Heavy Encoder: The heavy ViT image encoder must run on a GPU to achieve sub-second latency, limiting full client-side execution on low-power devices.
Glossary
Zero-Shot Transfer
The ability of a pre-trained model to generalize to novel tasks or data distributions that were not seen during training, without any parameter updates.
Intersection over Union (IoU)
A standard metric for evaluating segmentation quality, computed as the area of overlap between the predicted mask and the ground truth divided by the area of union.
Vision Transformer (ViT)
An architecture that applies the Transformer self-attention mechanism directly to sequences of image patches, widely used for scaling visual representation learning.
Cite this work
@article{kirillov2023segany,
title={Segment Anything},
author={Kirillov, Alexander and Mintun, Eric and Ravi, Nikhila and Mao, Hanzi and Rolland, Chloe and Gustafson, Laura and Xiao, Tete and Whitehead, Spencer and Berg, Alexander C. and Lo, Wan-Yen and Doll{\'a}r, Piotr and Girshick, Ross},
journal={arXiv:2304.02643},
year={2023}
}