SGLang

Implements a radix-tree based prefix cache that deduplicates and reuses KV cache across requests with shared prefixes, using a token-to-KV mapping system that tracks which tokens map to which cached KV states. The system automatically identifies common prefixes across concurrent requests and avoids redundant computation by serving cached KV pairs, reducing memory bandwidth and compute for subsequent tokens in the same prefix context.

Encodes output constraints (JSON schemas, regex patterns, grammar rules) as compressed finite state machines that guide token sampling during generation. The FSM is compiled from constraint specifications and integrated into the sampling pipeline, restricting logits to only tokens that maintain valid state transitions, ensuring generated output conforms to the schema without post-hoc validation or rejection sampling.

Exposes a gRPC server interface for high-performance client-server communication with support for streaming requests/responses and automatic request batching. The gRPC interface handles serialization, connection pooling, and multiplexing of concurrent requests, with lower latency and higher throughput than HTTP for high-frequency clients.

Orchestrates inference across multiple nodes using tensor parallelism, pipeline parallelism, and data parallelism with automatic load balancing. The system manages inter-node communication via NCCL or gRPC, distributes requests across nodes based on load, and handles node failures with graceful degradation. Supports both synchronous (all-reduce) and asynchronous (pipeline) execution patterns.

Implements a configurable sampling pipeline that processes logits through multiple stages: temperature scaling, top-k/top-p filtering, repetition penalties, length penalties, and custom constraints. Each stage is modular and can be enabled/disabled independently, with support for batch-level and token-level parameter variations. The pipeline integrates with the FSM-based constraint system for guaranteed valid outputs.

Implements a scheduler that separates prefill (processing prompt tokens) and decode (generating output tokens) into distinct phases, allowing different batch sizes and scheduling strategies for each. The scheduler batches prefill requests together, then schedules decode operations with higher priority to minimize latency. Supports continuous batching where new requests can be added to the decode queue without waiting for current requests to complete.

Provides a ModelConfig system that automatically detects model architecture (Llama, Qwen, DeepSeek, etc.) from HuggingFace model cards or manual specification, loads model weights with support for multiple formats (PyTorch, SafeTensors, GGUF), and handles architecture-specific optimizations. The system validates configuration compatibility and provides helpful error messages for unsupported models.

Provides a Python API for direct programmatic access to the SGLang inference engine, allowing applications to call the model without HTTP or gRPC overhead. The API exposes core functions like `generate()` and `chat()` that accept prompts and return generated text, with full control over generation parameters and access to internal state. This enables embedding SGLang directly in Python applications without network communication.

Automatically selects and orchestrates tensor parallelism (splitting model weights across GPUs), pipeline parallelism (splitting layers across GPUs), and expert parallelism (distributing MoE experts) based on model size, GPU count, and memory constraints. The system analyzes the model architecture, computes optimal partition strategies, and manages inter-GPU communication and synchronization transparently.

Pre-compiles model forward passes into CUDA graphs that capture GPU kernel launches and memory operations, then replays these graphs for each batch with dynamic shape handling. The system builds separate graphs for prefill and decode phases, caches graphs based on batch size and sequence length patterns, and reuses them across requests to eliminate CPU-GPU synchronization overhead and kernel launch latency.

Implements a hierarchical KV cache storage system (HiCache) that automatically tiers KV data across GPU VRAM, CPU RAM, and optional NVMe storage based on access patterns and memory pressure. The system monitors cache hit rates, predicts which KV states will be accessed, and proactively migrates data between tiers to minimize transfer latency while maximizing effective cache capacity.

Implements speculative decoding using EAGLE (a smaller draft model) that predicts multiple future tokens in parallel, which are then verified against the main model in a single forward pass. If verification succeeds, multiple tokens are accepted; if it fails, the draft is rejected and generation continues from the main model. The system integrates EAGLE predictions directly into the scheduling pipeline to minimize verification overhead.

Handles vision-language models (LLaVA, Qwen-VL, etc.) by preprocessing images into visual tokens, merging them with text tokens, and managing the combined sequence through the model. The system supports multiple image formats (JPEG, PNG, base64), resizes and patches images according to model requirements, and handles variable-length image sequences within batches.

Loads and applies LoRA (Low-Rank Adaptation) adapters to model weights at runtime without reloading the base model. The system maintains a registry of available adapters, patches model layers with adapter weights during forward passes, and supports switching between adapters across requests in the same batch. Adapters are merged into base weights for inference efficiency.

Supports multiple quantization schemes (FP8, FP4, INT8, MXFP4) with per-layer or per-channel quantization strategies. The system includes a quantization registry that maps quantization types to kernel implementations, handles quantization-aware training integration, and provides fallback kernels for unsupported hardware. Quantized models run with minimal accuracy loss while reducing memory footprint and increasing throughput.

Exposes an HTTP server with OpenAI API compatibility (chat completions, embeddings endpoints) that automatically formats conversations using model-specific chat templates. The system handles multi-turn conversations, system messages, and tool/function calling through standard OpenAI request/response formats, with automatic template selection based on model type.