sentence-transformers

Generates fixed-dimensional dense embeddings (typically 384-1024 dims) from text or images using transformer-based bi-encoder models that independently encode each input. The SentenceTransformer class wraps transformer models with pooling layers (mean, max, CLS token) to produce semantically meaningful vectors where cosine similarity directly reflects semantic relatedness. Supports batch processing with automatic padding and attention masking for variable-length inputs.

Generates sparse embeddings (vocabulary-sized dimensions, ~99% zeros) using the SparseEncoder class with models like SPLADE that learn to activate only relevant vocabulary dimensions. Combines neural matching signals with lexical interpretability by learning which vocabulary terms are relevant to each input. Outputs sparse tensors that can be indexed in traditional search engines (Elasticsearch, Solr) while maintaining neural ranking quality.

Automatically generates model cards (Hugging Face format) documenting model architecture, training data, performance metrics, and usage examples. Includes templates for different model types (SentenceTransformer, CrossEncoder, SparseEncoder) with sections for intended use, limitations, and bias/fairness considerations. Supports pushing model cards to Hugging Face Hub.

Supports memory-efficient training through gradient accumulation (simulating larger batch sizes without proportional memory increase), mixed precision training (float16 for forward/backward, float32 for loss), and distributed training across multiple GPUs/TPUs. Integrates with Hugging Face Trainer's optimization flags (gradient_checkpointing, fp16, deepspeed). Reduces memory footprint by 50-75% enabling training on smaller GPUs.

Implements multiple pooling strategies (mean pooling, max pooling, CLS token) to convert variable-length transformer outputs into fixed-size embeddings. Mean pooling averages all token embeddings (excluding padding), max pooling takes element-wise maximum, CLS pooling uses the [CLS] token embedding. Pooling layer is configurable and can be combined with other layers (normalization, projection). Handles attention masks automatically to exclude padding tokens.

Supports model quantization and optimization techniques (int8, fp16, distillation) to reduce model size and inference latency while maintaining embedding quality. Enables deployment on resource-constrained devices (mobile, edge) and reduces GPU memory requirements for large-scale indexing.

The CrossEncoder class jointly encodes text pairs to produce similarity scores, using a single transformer that processes concatenated inputs [CLS] text1 [SEP] text2 [SEP]. Outputs scalar scores (0-1 for classification, unbounded for regression) representing pair relevance. Designed for reranking retrieved candidates or classifying text pairs, with specialized loss functions (MarginMSELoss, CosineSimilarityLoss) optimized for ranking tasks.

Provides a modular training framework with 15+ loss functions (ContrastiveLoss, MultipleNegativesRankingLoss, MarginMSELoss, CosineSimilarityLoss, etc.) that can be combined and weighted for training custom embedding models. Each loss function is optimized for specific tasks: contrastive learning for similarity, triplet losses for ranking, margin-based losses for hard negatives. The SentenceTransformerTrainer class integrates with Hugging Face Trainer, supporting distributed training, mixed precision, and gradient accumulation.

Supports training and inference on multimodal models that jointly embed text and images into a shared vector space using dual-encoder architectures (separate text and image encoders with shared projection). Models like CLIP-based variants learn aligned representations where semantically related text and images have similar embeddings. Handles image preprocessing (resizing, normalization) and text tokenization automatically.

Computes similarity between embeddings using multiple metrics (cosine similarity, dot product, Euclidean distance, Manhattan distance) with vectorized implementations for efficient batch computation. The util module provides functions like semantic_search() that find top-k most similar embeddings using FAISS or brute-force methods, and paraphrase_mining() that identifies semantically similar sentence pairs within a corpus. Supports both normalized and unnormalized embeddings.

Provides evaluator classes (SentenceTransformerEvaluator, SparseEvaluator, CrossEncoderEvaluator) that compute ranking metrics (NDCG, MRR, MAP, Recall@k) and clustering metrics (accuracy, normalized mutual information) during training. Integrates with Hugging Face Trainer callbacks to log metrics at each epoch. Supports NanoBEIR benchmark for standardized evaluation across 35+ retrieval datasets.

Handles variable-length input sequences by automatically padding to the longest sequence in a batch and applying attention masks to prevent padding tokens from influencing embeddings. Supports batch processing with configurable batch sizes and automatic device placement (CPU/GPU). Includes show_progress_bar option for monitoring inference on large datasets. Tokenization and padding are handled internally via the underlying transformer model.

Loads pre-trained models directly from Hugging Face Hub (15,000+ models) using SentenceTransformer.from_pretrained() with automatic caching to ~/.cache/huggingface/. Supports loading from local paths, custom model cards, and automatic model selection based on task (e.g., 'all-MiniLM-L6-v2' for general semantic search). Handles model versioning and revision selection.

Provides utilities to combine dense retrieval (SentenceTransformer) with cross-encoder reranking in a two-stage pipeline: first stage retrieves top-k candidates using fast embedding similarity, second stage reranks using accurate cross-encoder scores. The semantic_search() function handles retrieval, and CrossEncoder.predict() handles reranking. Supports FAISS indexing for efficient retrieval on large corpora.