Segment Anything 2

Segments objects in static images using interactive point clicks or bounding box prompts, processed through a vision transformer image encoder that extracts dense feature maps, followed by a mask decoder that generates binary segmentation masks. The system uses a two-stage architecture where prompts are embedded and fused with image features via cross-attention mechanisms to produce precise object boundaries without requiring model retraining.

Generates segmentation masks for all salient objects in an image without user prompts by systematically sampling grid-based point prompts across the image and aggregating predictions through non-maximum suppression. The SAM2AutomaticMaskGenerator class orchestrates this process, using the image segmentation predictor to generate candidate masks at multiple scales and confidence thresholds, then deduplicates overlapping masks to produce a comprehensive segmentation map.

Generalizes to segment arbitrary object categories and visual domains without task-specific training, leveraging pre-training on diverse image datasets (SA-1B with 1.1B masks across 11M images). The model learns category-agnostic segmentation patterns through prompt-based learning, enabling segmentation of objects never seen during training. Generalization is enabled by the vision transformer's global receptive field and the prompt-based architecture that decouples object recognition from segmentation.

Propagates segmentation masks across video frames using predicted masks as implicit prompts, with confidence-based filtering to suppress low-confidence predictions and prevent error accumulation. The system computes confidence scores per frame based on prediction uncertainty, allowing downstream applications to filter unreliable masks or trigger re-prompting. Confidence filtering prevents cascading errors where a low-quality mask in frame N propagates to frame N+1.

Segments and tracks objects across video frames using a memory-augmented transformer architecture that maintains a streaming buffer of past frame embeddings and attention states. The SAM2VideoPredictor processes frames sequentially, encoding each frame through the vision transformer, fusing current frame features with historical memory via cross-attention mechanisms, and propagating object masks forward through time. Memory is selectively updated based on frame importance, enabling real-time processing without storing entire video histories.

Extends video segmentation to simultaneously track and segment multiple distinct objects across frames by maintaining separate mask predictions and memory states for each object. The system processes each object's trajectory independently through the video, allowing different objects to be prompted at different frames and tracked with object-specific temporal consistency. Mask propagation uses the previous frame's predicted mask as an implicit prompt for the next frame, creating a feedback loop that refines segmentation over time.

Provides a performance-optimized video predictor (SAM2VideoPredictorVOS) that applies PyTorch's torch.compile JIT compilation to the video segmentation pipeline, reducing memory overhead and accelerating frame processing. The VOS (Video Object Segmentation) variant specializes the streaming memory architecture for single-object tracking scenarios, eliminating multi-object overhead and enabling real-time inference on consumer GPUs. Compilation traces the attention and memory update operations, fusing them into optimized CUDA kernels.

Encodes input images through a hierarchical vision transformer (ViT) backbone that extracts multi-scale dense feature representations, processing images at multiple resolution levels to capture both semantic and fine-grained spatial information. The encoder produces feature pyramids with skip connections, enabling the mask decoder to access features at different scales for precise boundary localization. The architecture supports variable input resolutions by using patch-based tokenization and adaptive positional embeddings.

Refines segmentation masks through multiple decoder iterations that fuse user prompts (points, boxes, masks) with image features via cross-attention mechanisms. Each iteration updates the mask prediction by computing attention weights between prompt embeddings and image features, allowing the decoder to focus on relevant image regions and iteratively correct mask boundaries. The architecture supports mixed prompt types (e.g., combining point and box prompts) in a single forward pass through unified embedding and attention operations.

Provides four pre-trained model checkpoints (Tiny 38.9M, Small 46M, Base-Plus 80.8M, Large 224.4M parameters) with documented performance-accuracy tradeoffs, enabling developers to select variants based on deployment constraints. Each variant uses the same architecture but with different transformer depths and embedding dimensions, allowing inference speed to range from ~91 FPS (Tiny) to ~40 FPS (Large). Model selection is decoupled from application code, enabling runtime switching without code changes.

Integrates with Hugging Face Hub for seamless model checkpoint distribution, versioning, and community sharing. Models are loaded via a unified interface that automatically downloads checkpoints from the Hub, caches them locally, and manages version compatibility. The integration enables reproducible model loading across environments and facilitates community contributions of fine-tuned variants without requiring GitHub commits.

Processes multiple images or video frames in batches with automatic resolution normalization and padding, enabling efficient GPU utilization across diverse input dimensions. The system pads images to a common resolution within each batch, processes them through the vision transformer, and crops outputs back to original dimensions. Batch processing is transparent to the API — single-image and batch APIs are identical, with batching handled internally.