colbert-ai

Encodes documents as matrices of token-level embeddings rather than single vectors, using a fine-tuned BERT backbone to capture rich contextual information for each token. The encoder processes documents through the BERT transformer stack, producing a [num_tokens, embedding_dim] matrix per document that preserves fine-grained semantic relationships. This matrix representation enables late-interaction matching where query tokens can interact with individual document tokens rather than comparing aggregate vectors.

Implements efficient maximum similarity matching between query and document token embeddings using a specialized MaxSim operation that computes the maximum cosine similarity for each query token across all document tokens, then aggregates these maxima. This operation is implemented with CUDA kernels and optimized tensor operations to achieve sub-millisecond latency per query-document pair. The late-interaction design defers similarity computation until search time rather than pre-computing fixed document representations, enabling dynamic query-specific matching.

Implements performance-critical operations as custom CUDA kernels and optimized PyTorch operations, including MaxSim computation, embedding compression, and similarity aggregation. These kernels are fused to minimize memory bandwidth and kernel launch overhead, achieving 10-100x speedup over naive PyTorch implementations. Mixed-precision computation (FP16) is used throughout to reduce memory usage and increase throughput on modern GPUs.

Manages saving and loading of trained model checkpoints, including model weights, configuration, and training metadata. The checkpoint system saves checkpoints at regular intervals during training, tracks best checkpoints based on validation metrics, and enables resuming training from checkpoints. Checkpoints include model state dict, optimizer state, learning rate scheduler state, and training configuration for full reproducibility.

Enables training across multiple GPUs using PyTorch's distributed data parallelism, where each GPU processes a different batch of data and gradients are synchronized across GPUs. The distributed training setup handles gradient synchronization, loss aggregation, and checkpoint saving across processes. Training speed scales approximately linearly with number of GPUs (with some overhead for synchronization).

Processes large document collections across multiple GPUs and machines using a distributed indexing pipeline that encodes documents in batches, compresses token embeddings using product quantization or other compression schemes, and stores compressed representations in an inverted index structure. The pipeline manages memory efficiently by streaming documents through the encoder, compressing embeddings on-the-fly, and writing compressed vectors to disk in sharded index files. Configuration system allows tuning of batch sizes, compression rates, and number of indexing processes.

Retrieves candidate documents for a query using approximate nearest neighbor (ANN) search over compressed document embeddings, typically implemented with FAISS or similar ANN libraries. The system builds an ANN index over the compressed document embeddings during indexing, then uses the query embedding to retrieve top-k candidates (typically 1000-10000) in milliseconds. These candidates are then re-ranked using exact MaxSim computation to produce final results. The ANN search trades small precision loss for dramatic latency improvements, enabling sub-100ms end-to-end query latency.

Trains the ColBERT model end-to-end using contrastive learning objectives on query-document training pairs, where positive pairs are relevant documents and negative pairs are non-relevant documents. The trainer implements in-batch negatives, hard negative mining, and other techniques to improve training efficiency. Training uses mixed-precision computation (FP16) and gradient accumulation to fit large batch sizes on available GPUs. The trainer manages checkpoint saving, learning rate scheduling, and evaluation on validation sets during training.

Allows adding, removing, or updating documents in an existing index without reindexing the entire collection. The IndexUpdater component manages delta operations by encoding new documents, compressing them with the same compression scheme as the original index, and merging them into the index structure. For deletions, it marks documents as deleted in metadata without physically removing them. For updates, it re-encodes the document and replaces its compressed representation. This capability enables continuous index maintenance for evolving document collections.

Encodes search queries into token-level embedding matrices using the same BERT encoder as documents, producing a [num_query_tokens, embedding_dim] matrix. The query encoder applies the same tokenization and contextualization as the document encoder, ensuring compatible representations for MaxSim matching. Query encoding is fast (typically < 10ms) because queries are short, making it practical to encode queries at search time rather than pre-computing them.

Processes multiple queries in a single batch operation, encoding queries together and computing similarities against the index in vectorized operations. The batch search implementation uses PyTorch's batched matrix operations to compute MaxSim for all query-document pairs simultaneously, achieving higher throughput than sequential per-query search. Batch sizes are configurable and limited by GPU memory; typical batch sizes are 32-128 queries.

Provides a hierarchical configuration system that manages settings for model architecture, training hyperparameters, indexing parameters, and search settings. Configuration can be specified via Python objects, YAML files, or command-line arguments, with clear precedence rules for overrides. The configuration system validates settings, provides defaults, and enables reproducible experiments by capturing all hyperparameters in configuration files.

Computes standard information retrieval evaluation metrics including Mean Reciprocal Rank (MRR), Normalized Discounted Cumulative Gain (NDCG), Recall@k, and Mean Average Precision (MAP) on ranked retrieval results. The evaluation module compares predicted rankings against ground truth relevance judgments and aggregates metrics across queries. Metrics are computed efficiently using vectorized operations and support both binary and graded relevance judgments.