@memberjunction/ai-vectordb

Stores and retrieves high-dimensional vector embeddings with semantic search capabilities, enabling similarity-based document matching and RAG workflows. The module abstracts vector database operations through a provider-agnostic interface that supports multiple backend implementations (Pinecone, Weaviate, Milvus, etc.), allowing developers to swap vector stores without changing application code. Implements efficient indexing and querying patterns optimized for LLM context augmentation.

Executes semantic similarity search over document collections by converting queries to embeddings and ranking results by cosine distance or other similarity metrics. Implements query expansion and result filtering patterns to improve relevance, with configurable ranking strategies that can incorporate metadata filtering, recency weighting, or custom scoring functions. Designed to power LLM context retrieval with relevance-aware result ordering.

Manages the complete lifecycle of embeddings including creation, storage, updates, and deletion with consistency guarantees across vector database backends. Provides batch operations for efficient bulk embedding processing, handles embedding versioning when underlying models change, and maintains metadata synchronization between embeddings and source documents. Implements idempotent operations to prevent duplicate embeddings and supports incremental indexing for large document collections.

Abstracts away provider-specific vector database APIs through a unified interface that normalizes operations across Pinecone, Weaviate, Milvus, Qdrant, and other backends. Handles provider-specific configuration, connection pooling, and error handling transparently, allowing applications to switch providers by changing configuration without code changes. Implements provider capability detection to gracefully degrade features when backends don't support certain operations (e.g., metadata filtering, hybrid search).

Enables filtering vector search results by document metadata (tags, categories, dates, custom fields) while maintaining semantic relevance ranking. Implements metadata indexing alongside vector indexes to support efficient combined queries, with support for range queries, exact matches, and set membership operations. Allows composition of multiple metadata filters with AND/OR logic to narrow result sets before or after vector similarity ranking.

Orchestrates the complete RAG pipeline: query embedding, semantic retrieval, result ranking, and context assembly for LLM prompts. Handles automatic query preprocessing (normalization, expansion), implements configurable retrieval strategies (top-k, threshold-based, diversity sampling), and formats retrieved documents into structured context blocks suitable for LLM consumption. Provides hooks for custom ranking, filtering, and context formatting to adapt to domain-specific requirements.

Integrates with multiple embedding model providers (OpenAI, Hugging Face, local models) and caches embeddings to avoid redundant API calls and reduce costs. Implements embedding cache with configurable TTL and invalidation strategies, handles model versioning to track which model generated each embedding, and provides fallback mechanisms when primary embedding service is unavailable. Supports both API-based and local embedding models with automatic format normalization.

Implements multiple vector similarity metrics (cosine similarity, Euclidean distance, dot product, Manhattan distance) with optimized computation for high-dimensional vectors. Provides configurable distance metrics per query, handles vector normalization and dimension validation, and supports approximate nearest neighbor search for performance optimization on large collections. Includes utilities for similarity score interpretation and threshold-based result filtering.

Implements configurable document chunking strategies (fixed-size, semantic, sliding window) to break large documents into embeddable units while preserving context. Handles chunk overlap configuration, metadata propagation from parent documents to chunks, and chunk reassembly for context reconstruction. Supports adaptive chunking based on document structure (paragraphs, sentences) and provides utilities for chunk quality assessment (length validation, content filtering).