llama.cpp

Executes LLM inference on GGUF-quantized models through a pluggable backend system supporting CPU (AVX/AVX2/AVX512/NEON), GPU (CUDA/Metal/Vulkan/HIP/SYCL), and specialized hardware (RISC-V, AMX). The backend registration pattern allows runtime selection of compute targets without recompilation, with automatic fallback to CPU if GPU unavailable. Quantization is applied at model load time via GGML's tensor library, reducing memory footprint by 4-8x while maintaining inference quality through techniques like Q4_K_M and Q5_K_M.

Parses GGUF (GGML Universal Format) binary files containing quantized weights, metadata, and vocabulary through a custom file format reader that supports memory-mapped access for zero-copy weight loading. The format encodes tensor shapes, quantization types (Q4_K, Q5_K, Q6_K, etc.), and model hyperparameters in a structured header, enabling lazy loading of weights on-demand. Supports model conversion from HuggingFace safetensors/PyTorch via Python conversion scripts that apply quantization during serialization.

Enables parameter-efficient fine-tuning via Low-Rank Adaptation (LoRA) adapters that add trainable low-rank matrices to model weights without modifying base weights. The system loads base model from GGUF, applies LoRA adapters at inference time by computing W = W_base + A × B^T (where A, B are low-rank matrices), and supports multiple adapters with weighted combination. Adapters are stored separately from base model, enabling easy sharing and composition.

Extends text-only inference to support image inputs via libmtmd (multimodal) integration, which encodes images using vision transformers (e.g., CLIP) and fuses visual embeddings with text tokens. The system handles image preprocessing (resizing, normalization), vision model inference, and embedding concatenation before passing to LLM. Supports multiple image formats (PNG, JPEG, WebP) and variable image sizes with automatic padding.

Generates spoken audio from text using integrated TTS models (e.g., Piper, XTTS) that support voice cloning via speaker embeddings. The system converts text to phonemes, synthesizes audio waveforms, and applies voice characteristics from reference audio. Supports multiple languages, voice styles, and real-time streaming output via audio chunks.

Converts models from HuggingFace formats (safetensors, PyTorch) to GGUF with configurable quantization levels (Q4_K_M, Q5_K_S, Q6_K, etc.) via Python conversion scripts. The system reads model architecture from HuggingFace config.json, maps weights to GGML tensor operations, applies quantization during serialization, and validates output integrity. Supports 100+ model architectures (Llama, Mistral, Phi, Qwen, etc.) with architecture-specific handling.

Manages multiple models in a single server instance with dynamic routing based on request metadata (model name, task type, resource constraints). The system loads models on-demand, caches hot models in memory, and switches between models based on request parameters. Supports load balancing across multiple model instances and automatic offloading of idle models to disk to free VRAM.

Processes multiple inference requests in parallel through a batch scheduling system that groups tokens into compute-efficient batches, with dynamic KV (key-value) cache allocation that reuses cache slots across sequences. The pipeline uses a llama_batch structure to encode token positions, sequence IDs, and logits flags, enabling efficient multi-sequence inference where shorter sequences don't block longer ones. KV cache is allocated per-sequence and can be pruned (via kv_keep_only_active) to remove inactive sequences, reducing memory overhead in long-running services.

Enforces structured output by parsing EBNF grammar rules and filtering the sampling distribution to only allow tokens that maintain grammar validity. The system builds a finite-state automaton from grammar rules, tracks parser state across tokens, and masks logits for invalid next tokens before sampling. Supports JSON schema grammars (via gbnf format) for type-safe structured generation, enabling deterministic parsing of model outputs without post-hoc validation.

Accelerates token generation by running a smaller draft model to predict multiple tokens ahead, then verifying predictions with the main model in parallel. The system generates k draft tokens, encodes them into a batch, runs the main model once to validate all k positions, and accepts/rejects tokens based on probability matching. Reduces main model invocations from N to ~N/k, where k is typically 4-8, achieving 2-3x speedup with minimal quality loss.

Parses and applies chat templates (Jinja2-based format strings) to convert structured message lists into properly formatted prompts for different model architectures. The system reads template metadata from GGUF model headers, applies variable substitution for roles (user, assistant, system), handles special tokens (BOS, EOS, padding), and manages conversation history formatting. Supports model-specific conventions (e.g., Llama 2's [INST] tags, Mistral's [INST] with system prompt handling) without manual prompt engineering.

Distributes model layers across multiple GPUs by splitting the computation graph at layer boundaries, with each GPU computing its assigned layers and passing activations to the next GPU. The system uses a layer-to-device mapping (via ggml_backend_sched) to schedule tensor operations, managing inter-GPU communication via PCIe or NVLink. Supports both pipeline parallelism (sequential layer execution) and tensor parallelism (layer sharding) depending on hardware topology.

Implements multiple sampling strategies to control generation diversity by transforming logits before sampling. Temperature scaling adjusts logit magnitudes (lower = more deterministic), top-k filters to k highest-probability tokens, top-p (nucleus sampling) filters to cumulative probability threshold, and min-p enforces minimum probability relative to max logit. Strategies can be combined (e.g., top-k + top-p) and applied per-token, enabling fine-grained control over output randomness.

Converts text to token IDs using model-specific vocabularies (typically BPE or SentencePiece) loaded from GGUF metadata, with special handling for control tokens (BOS, EOS, padding), role markers, and tool tokens. The system maintains a token-to-ID mapping, supports both encoding (text → tokens) and decoding (tokens → text), and handles edge cases like unknown tokens and multi-byte UTF-8 sequences. Vocabulary size varies by model (32K-128K tokens) and directly impacts model architecture.

Exposes llama.cpp inference through a REST API (llama-server) with OpenAI-compatible endpoints (/v1/chat/completions, /v1/completions, /v1/embeddings). The server manages model loading, request queuing, batch scheduling, and response streaming via Server-Sent Events (SSE). Supports concurrent requests with automatic batching, slot-based KV cache management for multi-user scenarios, and metrics collection (tokens/sec, latency percentiles).