CLIP

Classifies images into arbitrary categories without training by encoding images and text descriptions into a shared embedding space, then computing cosine similarity between image embeddings and text embeddings to determine the best matching class. The dual-encoder architecture (separate image and text encoders) projects both modalities into the same vector space where semantically related concepts cluster together, enabling direct comparison without fine-tuning on target classes.

Computes similarity scores between images and text by encoding both into a shared embedding space and calculating cosine similarity between their feature vectors. The model uses contrastive loss training to align image and text embeddings such that matching pairs have high similarity and mismatched pairs have low similarity. This enables ranking images by relevance to text queries or vice versa.

Extracts fixed-size feature vectors (embeddings) from images and text by passing them through trained encoders (ResNet/ViT for images, Transformer for text) and projecting outputs into a shared embedding space. These embeddings capture semantic information and can be used for downstream tasks like clustering, nearest-neighbor search, or as input to other models. The embedding space is learned via contrastive pre-training to align related images and text.

Provides 9 pre-trained model variants with different architectures (ResNet-50/101 vs Vision Transformer) and parameter counts (50M to 400M) to enable trade-offs between accuracy, speed, and memory. Models are loaded via clip.load(name, device) which downloads from OpenAI's Azure endpoint and places on specified device (CPU/GPU). Each variant has different input image sizes (224px to 448px) and embedding dimensions, allowing users to select based on latency/accuracy requirements.

Tokenizes text inputs into fixed-length token sequences (default 77 tokens) using a custom byte-pair encoding (BPE) tokenizer trained on the pre-training corpus. The clip.tokenize() function handles padding/truncation to context length and returns integer token IDs that can be passed to the text encoder. Supports batch tokenization and preserves token-to-character mappings for interpretability.

Encodes batches of images into embeddings by applying preprocessing (resizing, normalization) and passing through the image encoder (ResNet or ViT). The preprocessing transform is returned by clip.load() and handles ImageNet normalization (mean=[0.48145466, 0.4578275, 0.40821073], std=[0.26862954, 0.26130258, 0.27577711]). Supports automatic device placement (CPU/GPU) and batching for efficiency, with typical throughput of 100-500 images/second depending on model size and hardware.

Implements a shared embedding space where images and text are projected such that matching pairs have high cosine similarity and mismatched pairs have low similarity. This alignment is learned via contrastive pre-training (InfoNCE loss) on 400M image-text pairs, enabling the model to understand semantic relationships between visual and textual concepts without explicit supervision on target tasks. The shared space enables zero-shot transfer because new classes can be described in text and compared directly to image embeddings.

Provides two families of image encoders: ResNet variants (RN50, RN101, RN50x4, RN50x16, RN50x64) and Vision Transformer variants (ViT-B/32, ViT-B/16, ViT-L/14, ViT-L/14@336px). ResNets use convolutional layers with residual connections, while ViTs use multi-head self-attention on image patches. Both are trained with the same contrastive objective and produce embeddings in the same shared space, but differ in accuracy, speed, and memory characteristics. Users select architecture via clip.load(name) without code changes.

Loads pre-trained CLIP models from OpenAI's Azure endpoint via clip.load(name, device, jit) and caches them locally to avoid repeated downloads. Handles device placement (CPU/GPU) automatically and optionally applies JIT compilation for faster inference. The function returns both the model object and a preprocessing transform, enabling immediate use without additional setup. Models are downloaded on first use and cached in ~/.cache/clip/ by default.

While CLIP itself is pre-trained and not fine-tuned in the standard library, the architecture enables contrastive loss training on custom image-text datasets. The model can be adapted by computing embeddings for image-text pairs and applying InfoNCE loss (or similar contrastive objectives) to align custom data in the shared embedding space. This requires custom training code but leverages CLIP's pre-trained encoders as initialization, enabling efficient domain adaptation with smaller datasets.