nexa-sdk

Executes large language models locally across CPU, GPU, and NPU hardware through a layered architecture that abstracts hardware differences via a plugin system. The Go SDK provides type-safe interfaces (Create/Destroy lifecycle) that route inference requests through CGo bindings to C/C++ hardware plugins, enabling day-0 support for models like GPT-OSS, Granite-4, Qwen-3, and Llama-3 without cloud dependencies. Model formats (GGUF, MLX, NEXA) are handled by format-specific plugins that optimize for target hardware capabilities.

Processes images and text together through VLM models (Qwen-3-VL, etc.) using a unified Go SDK interface that handles image encoding, tokenization, and vision-specific hardware optimizations. The VLM plugin system manages image preprocessing (resizing, normalization) and routes vision tokens through specialized hardware paths (GPU tensor cores for image encoding, NPU for attention). Supports batch image processing and maintains image context across multi-turn conversations.

Provides Python bindings to the Go SDK through a wrapper layer that exposes model classes (LLM, VLM, Embedder, etc.) with Create/Destroy lifecycle management. Supports both synchronous and asynchronous inference via asyncio, enabling concurrent model execution. Implements model caching and keepalive mechanisms to avoid reloading models between requests. Type hints and docstrings enable IDE autocomplete and documentation.

Provides Android-specific bindings to the Nexa inference engine through JNI (Java Native Interface) bridges. Implements model lifecycle management (Create/Destroy) with automatic cleanup on activity destruction. Supports both synchronous and asynchronous inference via Android's Executor framework. Handles Android-specific constraints (memory pressure, background execution, battery optimization) through lifecycle-aware components.

Provides iOS-specific bindings to the Nexa inference engine through Swift/Objective-C bridges. Implements Metal GPU acceleration for inference on Apple devices, leveraging GPU compute shaders for matrix operations. Supports iOS app extensions (Siri, keyboard, share) enabling inference in restricted execution contexts. Implements background task management for long-running inference with proper battery optimization.

Provides Docker images and containerization support for deploying Nexa on Linux servers and IoT devices. Supports both Arm64 (Raspberry Pi, Jetson, etc.) and x86-64 architectures with hardware-specific optimizations (CUDA for x86 GPU, NEON for Arm64 CPU). Implements multi-stage builds to minimize image size and includes pre-configured models for common use cases. Supports Docker Compose for orchestrating multi-model inference services.

Implements structured function calling through a schema-based tool registry that defines function signatures as JSON schemas. Supports OpenAI and Anthropic function-calling protocols natively, enabling agents to invoke external tools with type-safe arguments. The server middleware validates function calls against schemas, handles tool execution, and formats responses back to the model. Supports both synchronous tool execution and async tool chains.

Exposes local inference models via REST API endpoints that mirror OpenAI's chat completion and embedding APIs, enabling drop-in replacement of cloud LLM services. The server implements streaming responses (Server-Sent Events), function calling via schema-based function registry with native bindings for OpenAI/Anthropic APIs, and middleware for request validation, rate limiting, and response formatting. Built on Go HTTP server with configurable port and model routing.

Manages model lifecycle (discovery, download, caching, updates) across multiple model repositories (Hugging Face, ModelScope, Volces, S3, local filesystem) through a pluggable model hub system. Implements intelligent caching with file locking to prevent concurrent downloads, manifest tracking for version management, and automatic model updates. The store manager (runner/internal/store/) handles disk space management, model validation, and atomic file operations to ensure consistency across platform crashes.

Generates dense vector embeddings for text using embedding models (e.g., BGE, ONNX-based embedders) through the embedder interface (runner/nexa-sdk/embedder.go). Embeddings are computed locally on GPU/NPU for privacy, supporting batch processing to amortize inference overhead. Integrates with vector databases via standard embedding output format (float32 arrays), enabling semantic search, similarity matching, and RAG pipeline construction without external embedding services.

Implements cross-encoder reranking (runner/nexa-sdk/reranker.go) to refine retrieval results by scoring query-document pairs jointly, improving RAG pipeline precision. Rerankers take query and candidate documents as input, compute relevance scores, and return ranked results. Operates on GPU/NPU for efficient batch scoring of large result sets, supporting both pointwise (single score per document) and pairwise (comparative scoring) reranking strategies.

Generates images from text prompts using Stable Diffusion and compatible latent diffusion models through a dedicated image generation plugin. Implements the full diffusion pipeline (text encoding, latent diffusion, VAE decoding) with hardware-specific optimizations for GPU/NPU. Supports various sampling strategies (DDPM, DDIM, Euler), LoRA adapters for style transfer, and negative prompts for quality control. Outputs PNG/JPEG images with configurable resolution and quality parameters.

Converts text to natural-sounding speech using TTS models through the audio processing plugin system. Implements streaming audio generation where speech is synthesized incrementally and output as audio chunks (WAV, MP3), enabling real-time playback without waiting for full synthesis. Supports multiple voices, speaking rates, and prosody control. Hardware acceleration on GPU/NPU speeds up mel-spectrogram generation and vocoder inference.

Transcribes audio to text using ASR models (Whisper, etc.) through the audio processing plugin system. Supports streaming transcription where audio chunks are processed incrementally, enabling real-time speech-to-text without waiting for full audio. Implements voice activity detection (VAD) to skip silence, reducing computation. Outputs text with optional timestamps and confidence scores. Hardware acceleration on GPU/NPU speeds up acoustic model inference.

Provides a comprehensive CLI (runner/cmd/nexa-cli/) for model discovery, download, inference, and server management. Implements an interactive REPL mode for testing models with multi-turn conversations, model listing/info commands, and server startup. The CLI routes commands through the core orchestration layer (Layer 2) which parses arguments and dispatches to appropriate Go SDK methods. Supports both one-shot inference (nexa run model 'prompt') and interactive sessions (nexa infer model).