Llamafile

Packages LLMs as self-contained executable files by combining llama.cpp inference engine with Cosmopolitan Libc, enabling distribution of model weights and binary code in a single file that executes on Windows, macOS, and Linux without installation. The file is structured as a polyglot shell script containing AMD64 and ARM64 binaries that auto-detect and execute the appropriate architecture.

Executes LLM inference using GGML (Generalized Matrix Language) tensor library for efficient matrix operations, supporting multiple quantization formats (Q4, Q5, Q8, etc.) that reduce model size and memory footprint while maintaining inference quality. The system allocates tensors via ggml-alloc.c with automatic memory pooling and reuses KV (Key-Value) cache across inference steps to minimize redundant computation.

Converts full-precision LLM models to GGUF quantized formats (Q4, Q5, Q8, etc.) via quantize tool, reducing model size 4-8x while maintaining inference quality. Supports importance matrix (imatrix) calculation for optimal quantization, allowing selective quantization of important layers with higher precision.

Detects host CPU architecture (x86-64, ARM64) at runtime and automatically selects appropriate binary code path from polyglot executable, enabling single file to run on Windows, macOS, and Linux without manual architecture selection. File structure embeds both AMD64 and ARM64 binaries as shell script with embedded ELF/Mach-O headers.

Manages the model's context window (maximum sequence length) and optimizes KV cache allocation to fit within available VRAM. Implements sliding window attention for models supporting it, allowing inference on sequences longer than model's training context while maintaining constant memory usage. Tracks token positions and manages cache eviction when context exceeds available memory.

Processes both text and images by encoding images through a CLIP image encoder into embeddings, projecting those embeddings into the LLM's token embedding space via a multimodal projector, and combining projected embeddings with text tokens for unified inference. Supports models like LLaVA that can answer questions about images or describe visual content.

Provides CLI interface for text generation with fine-grained control over sampling methods (temperature, top-k, top-p, min-p), token limits, and stopping conditions. Tokenizes input via llama_tokenize(), processes tokens through llama_decode() to generate logits, applies sampling via llama_sampling_sample() to select next tokens, and repeats until stopping condition is met or max tokens reached.

Launches an embedded HTTP server that exposes REST API endpoints compatible with OpenAI's chat completion and completion APIs, enabling integration with existing LLM client libraries and applications. Server manages concurrent inference requests via slot management (allocating KV cache slots per request), handles streaming responses via Server-Sent Events (SSE), and provides web UI for interactive chat.

Manages multiple concurrent inference requests by allocating separate KV (Key-Value) cache slots to each request, preventing cache collisions and enabling parallel inference. Each slot maintains independent attention cache state, allowing the server to process multiple prompts simultaneously up to the limit of available VRAM and configured slot count.

Offloads tensor operations to NVIDIA GPUs via CUDA or AMD GPUs via ROCm, automatically detecting available hardware and routing matrix multiplications to GPU while keeping model weights in GPU memory. Build scripts (cuda.sh, rocm.sh) compile llamafile with GPU support, and runtime automatically selects GPU kernels for supported operations.

Optimizes tensor operations for CPU execution using SIMD instructions (AVX2 on x86-64, NEON on ARM), enabling efficient matrix multiplications without GPU. GGML kernels detect CPU capabilities at runtime and dispatch to optimized code paths, providing 2-4x speedup versus scalar operations.

Provides built-in web interface accessible via browser that enables interactive chat with the loaded model, file upload for multimodal inputs, and real-time streaming responses. UI communicates with the HTTP server via JavaScript, displaying responses as they stream via Server-Sent Events (SSE).

Integrates Whisper speech recognition model to transcribe audio input into text, which can then be processed by the LLM. Whisper model runs locally in the same process, converting audio files or streams into text tokens that feed into the LLM inference pipeline.