ruvector-onnx-embeddings-wasm

Compiles ONNX sentence-transformer models to WebAssembly with SIMD (Single Instruction Multiple Data) intrinsics for vectorized tensor operations, enabling native embedding inference across browsers, Cloudflare Workers, Deno, and Node.js without external ML runtime dependencies. Uses WASM linear memory for model weights and intermediate activations, with SIMD instructions for matrix multiplication and normalization operations to achieve near-native performance on CPU-bound embedding tasks.

Distributes embedding inference across multiple worker threads (Node.js Worker Threads, Web Workers in browsers, Deno workers) to parallelize computation on multi-core systems. Each worker maintains its own WASM module instance and embedding model state, processing disjoint batches of text independently and returning results via message passing, enabling linear throughput scaling with core count for large-scale embedding generation.

Loads ONNX model files (serialized protobuf format) into WASM memory, parses the computation graph (nodes, operators, tensor metadata), and initializes the WASM runtime with model weights and operator implementations. Supports lazy-loading of model weights from URLs or local files, with optional model quantization (int8, float16) to reduce memory footprint and improve inference speed on resource-constrained environments like browsers and edge workers.

Converts raw text input into token IDs using BPE (Byte-Pair Encoding) or WordPiece tokenization, applies special tokens (CLS, SEP, PAD), and generates attention masks required by transformer embedding models. Tokenization runs in WASM or JavaScript depending on performance requirements, with support for batch processing and configurable max sequence length with truncation/padding strategies.

Computes cosine similarity, Euclidean distance, and dot-product similarity between embedding vectors using SIMD-accelerated operations in WASM. Supports batch similarity computation (e.g., query embedding vs. document embeddings matrix), with optional GPU acceleration via WebGPU for large-scale similarity searches. Results are typically used for semantic search ranking, nearest-neighbor retrieval, and clustering tasks.

Caches computed embeddings in memory (LRU cache, IndexedDB for browsers) keyed by text hash, avoiding redundant embedding computation for repeated inputs. Supports cache invalidation strategies (TTL, size limits, manual clearing) and optional persistence to local storage or IndexedDB for cross-session reuse, reducing embedding latency from 50-500ms to <1ms for cached queries.

Automatically detects runtime environment (Node.js, browser, Deno, Cloudflare Workers) and selects appropriate WASM module variant, worker thread implementation, and I/O APIs. Provides unified JavaScript API across all runtimes, abstracting away platform-specific differences (e.g., Node.js fs module vs. browser fetch API, Worker Threads vs. Web Workers). Enables single codebase deployment to multiple targets without conditional compilation.

Provides integration points for Retrieval-Augmented Generation (RAG) workflows: embedding documents for indexing, storing embeddings in vector databases (Pinecone, Weaviate, Milvus, local vector stores), and retrieving top-K similar documents for LLM context. Includes utilities for document chunking, metadata attachment, and batch indexing to vector stores, enabling end-to-end RAG pipelines from raw documents to LLM-augmented responses.

Reduces ONNX model size through quantization (int8, float16) and pruning, enabling deployment to resource-constrained environments (browsers, edge workers) where full-precision models exceed memory budgets. Quantization typically reduces model size by 4x (float32 → int8) with minimal embedding quality loss (<2% cosine similarity degradation). Includes quantization-aware training support and post-training quantization with calibration data.

Implements dynamic batching for embedding inference, accumulating multiple embedding requests and processing them together to maximize GPU/CPU utilization and amortize model loading overhead. Includes configurable batch size limits, timeout-based batch flushing (e.g., flush after 100ms even if batch not full), and priority queue support for latency-sensitive requests. Enables high-throughput embedding generation (1000+ embeddings/sec) on multi-core systems.