weaviate

Implements Hierarchical Navigable Small World (HNSW) algorithm for sub-linear time complexity vector similarity search across high-dimensional embeddings. The implementation supports dynamic index construction with configurable M (max connections per node) and ef (search parameter) values, enabling tuning of recall vs latency tradeoffs. Search queries traverse the hierarchical graph structure to locate nearest neighbors without exhaustive comparison, returning results ranked by vector distance.

Executes multi-stage search pipelines that fuse vector similarity results with BM25 full-text search scores and apply WHERE-clause filtering on structured properties. The query executor (Traverser and Explorer patterns) orchestrates parallel vector and keyword index lookups, then merges ranked results using configurable fusion algorithms (RRF, weighted sum). Inverted index with delta-merger pattern enables incremental BM25 index updates without full rebuilds.

Provides backup/restore functionality with support for incremental snapshots (only changed data since last backup) and pluggable offload modules for storing backups in external storage (S3, GCS, Azure Blob). Backup process creates consistent snapshots across all shards using Raft consensus. Restore operation validates backup integrity and replays changes to restore cluster to specific point-in-time. Offload modules enable storing backups in cloud storage without local disk requirements.

Supports image objects with automatic vectorization using multi-modal embedding models (CLIP, etc.) that generate vectors from image content. Image search enables finding visually similar images by uploading query image or providing image URL. Vectorizer modules handle image download, preprocessing, and embedding generation. Supports both image-to-image search and text-to-image search using shared embedding space.

Exposes REST API with full OpenAPI 3.0 specification enabling auto-generated API documentation and client SDK generation. API endpoints cover CRUD operations, search, schema management, and cluster operations. OpenAPI spec is machine-readable, enabling API discovery and validation. Swagger UI provides interactive API exploration and testing. REST API supports both JSON request/response and streaming responses for large result sets.

Exposes Prometheus metrics for monitoring query latency, throughput, error rates, and resource utilization. Supports distributed tracing via OpenTelemetry, enabling end-to-end request tracing across services. Telemetry collection is configurable with sampling to reduce overhead. Metrics cover API layer (request counts, latencies), storage layer (index operations, disk I/O), and cluster operations (Raft consensus, replication).

Implements dynamic index selection that automatically chooses between HNSW (for large datasets) and flat index (for small datasets) based on shard size. Flat index performs exhaustive search without index structure, optimal for <10K vectors. HNSW index is automatically created when shard exceeds threshold. Dynamic switching enables optimal performance across dataset sizes without manual tuning. Index type can be explicitly configured if needed.

Partitions data across multiple shards (horizontal scaling) with each shard maintaining LSM-KV storage engine for durability. Raft consensus protocol coordinates writes across shard replicas, ensuring consistency guarantees (quorum-based acknowledgment). Shard routing layer automatically distributes objects by hash and replicates writes to configured replica count, with automatic failover when replicas become unavailable. Lazy-loader pattern defers shard initialization until first access.

Provides high-throughput batch write API that queues objects for asynchronous processing with configurable batch sizes and concurrency. Implements transactional semantics where entire batch succeeds or fails atomically, with per-object error reporting. Job queue distributes batch operations across worker threads, with backpressure handling to prevent memory exhaustion. Write path (shard_write_batch_objects.go) coordinates object insertion, vector index updates, and inverted index updates in single transaction.

Exposes GraphQL interface for querying objects with support for nested property selection, cross-object references, and aggregation functions (count, sum, mean, max, min). Query executor traverses object relationships defined in schema, fetching related objects in single query without N+1 round-trips. Aggregation pipeline computes statistics across result sets (e.g., average vector distance, object count by category).

Provides gRPC interface as alternative to REST/GraphQL with support for bidirectional streaming, enabling efficient bulk operations and real-time result streaming. Protocol buffers define strongly-typed message contracts with automatic code generation for multiple languages. Streaming reduces overhead vs request-response pattern, particularly for batch operations and large result sets. gRPC multiplexing over HTTP/2 enables connection reuse and header compression.

Module system allows plugging in external vectorizer implementations (OpenAI, Hugging Face, Cohere, etc.) to automatically generate embeddings for text properties during object creation. Vectorizer modules intercept write operations, extract text from specified properties, call external embedding API, and store resulting vectors. Supports custom vectorizer implementations via module interface, enabling proprietary embedding models. Caching layer reduces redundant API calls for duplicate text.

Module system supports plugging in generative models (LLMs) and reranking models to post-process search results. Generative modules take search results and generate synthetic content (summaries, answers, completions) using external LLM APIs. Reranker modules re-rank search results using cross-encoder models, improving relevance beyond vector similarity. Modules receive search context (query, results) and return enriched results with generated content or adjusted rankings.

Implements RBAC system with built-in roles (admin, editor, viewer) and custom role definitions with granular permissions across domains (collections, objects, backups). Permission model supports permission domains enabling fine-grained access control (e.g., read-only access to specific collections). Multi-tenancy support allows isolating data per tenant with tenant-specific RBAC policies. Authentication integrates with OIDC providers and API key-based auth.

Manages data schema (class definitions, properties, indexes) with Raft consensus ensuring all nodes have identical schema state. Schema changes (add/remove properties, modify indexes) are coordinated through Raft leader, preventing split-brain scenarios. Schema manager validates changes against existing data and coordinates index migrations. Supports schema versioning and deprecation tracking for backward compatibility.