name: segment-anything-model
description: Foundation model for image segmentation with zero-shot transfer. Use when you need to segment any object in images using points, boxes, or masks as prompts, or automatically generate all object masks in an image.
version: 1.0.0
author: Orchestra Research
license: MIT
dependencies: [segment-anything, transformers>=4.30.0, torch>=1.7.0]
metadata:
hermes:
tags: [Multimodal, Image Segmentation, Computer Vision, SAM, Zero-Shot]
Segment Anything Model (SAM)
Comprehensive guide to using Meta AI's Segment Anything Model for zero-shot image segmentation.
When to use SAM
Use SAM when:
- - Need to segment any object in images without task-specific training
- - Building interactive annotation tools with point/box prompts
- - Generating training data for other vision models
- - Need zero-shot transfer to new image domains
- - Building object detection/segmentation pipelines
- - Processing medical, satellite, or domain-specific images
- - Zero-shot segmentation: Works on any image domain without fine-tuning
- - Flexible prompts: Points, bounding boxes, or previous masks
- - Automatic segmentation: Generate all object masks automatically
- - High quality: Trained on 1.1 billion masks from 11 million images
- - Multiple model sizes: ViT-B (fastest), ViT-L, ViT-H (most accurate)
- - ONNX export: Deploy in browsers and edge devices
- - YOLO/Detectron2: For real-time object detection with classes
- - Mask2Former: For semantic/panoptic segmentation with categories
- - GroundingDINO + SAM: For text-prompted segmentation
- - SAM 2: For video segmentation tasks
- - Advanced Usage - Batching, fine-tuning, integration
- - Troubleshooting - Common issues and solutions
- - GitHub: https://github.com/facebookresearch/segment-anything
- - Paper: https://arxiv.org/abs/2304.02643
- - Demo: https://segment-anything.com
- - SAM 2 (Video): https://github.com/facebookresearch/segment-anything-2
- - HuggingFace: https://huggingface.co/facebook/sam-vit-huge
Key features:
Use alternatives instead:
Quick start
Installation
`bash
From GitHub
pip install git+https://github.com/facebookresearch/segment-anything.git
Optional dependencies
pip install opencv-python pycocotools matplotlib
Or use HuggingFace transformers
pip install transformers
`
Download checkpoints
`bash
ViT-H (largest, most accurate) - 2.4GB
wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth
ViT-L (medium) - 1.2GB
wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_l_0b3195.pth
ViT-B (smallest, fastest) - 375MB
wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_b_01ec64.pth
`
Basic usage with SamPredictor
`python
import numpy as np
from segment_anything import sam_model_registry, SamPredictor
Load model
sam = sam_model_registry"vit_h"
sam.to(device="cuda")
Create predictor
predictor = SamPredictor(sam)
Set image (computes embeddings once)
image = cv2.imread("image.jpg")
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
predictor.set_image(image)
Predict with point prompts
input_point = np.array([[500, 375]]) # (x, y) coordinates
input_label = np.array([1]) # 1 = foreground, 0 = background
masks, scores, logits = predictor.predict(
point_coords=input_point,
point_labels=input_label,
multimask_output=True # Returns 3 mask options
)
Select best mask
best_mask = masks[np.argmax(scores)]
`
HuggingFace Transformers
`python
import torch
from PIL import Image
from transformers import SamModel, SamProcessor
Load model and processor
model = SamModel.from_pretrained("facebook/sam-vit-huge")
processor = SamProcessor.from_pretrained("facebook/sam-vit-huge")
model.to("cuda")
Process image with point prompt
image = Image.open("image.jpg")
input_points = [[[450, 600]]] # Batch of points
inputs = processor(image, input_points=input_points, return_tensors="pt")
inputs = {k: v.to("cuda") for k, v in inputs.items()}
Generate masks
with torch.no_grad():
outputs = model(**inputs)
Post-process masks to original size
masks = processor.image_processor.post_process_masks(
outputs.pred_masks.cpu(),
inputs["original_sizes"].cpu(),
inputs["reshaped_input_sizes"].cpu()
)
`
Core concepts
Model architecture
`
SAM Architecture:
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Image Encoder │────▶│ Prompt Encoder │────▶│ Mask Decoder │
│ (ViT) │ │ (Points/Boxes) │ │ (Transformer) │
└─────────────────┘ └─────────────────┘ └─────────────────┘
│ │ │
Image Embeddings Prompt Embeddings Masks + IoU
(computed once) (per prompt) predictions
`
Model variants
| Model | Checkpoint | Size | Speed | Accuracy |
| ------- | ------------ | ------ | ------- | ---------- |
| ViT-H | vit_h | 2.4 GB | Slowest | Best |
| ViT-L | vit_l | 1.2 GB | Medium | Good |
| ViT-B | vit_b | 375 MB | Fastest | Good |
| Prompt | Description | Use Case | ||
| -------- | ------------- | ---------- | ||
| Point (foreground) | Click on object | Single object selection | ||
| Point (background) | Click outside object | Exclude regions | ||
| Bounding box | Rectangle around object | Larger objects | ||
| Previous mask | Low-res mask input | Iterative refinement | ||
| Issue | Solution | |||
| ------- | ---------- | |||
| Out of memory | Use ViT-B model, reduce image size | |||
| Slow inference | Use ViT-B, reduce points_per_side | |||
| Poor mask quality | Try different prompts, use box + points | |||
| Edge artifacts | Use stability_score filtering | |||
| Small objects missed | Increase points_per_side |