Lemonade by AMD: a fast and open source local LLM server using GPU and NPU

Executes large language model inference on AMD GPUs using the ROCm (Radeon Open Compute) platform, enabling hardware-accelerated tensor operations without cloud dependencies. The server implements GPU memory management, kernel scheduling, and compute graph optimization specific to AMD RDNA/CDNA architectures, allowing models to run at native GPU speeds with automatic batching and memory pooling.

Distributes inference workloads across integrated NPUs (found in AMD Ryzen AI and similar processors) alongside GPU/CPU resources using a heterogeneous scheduler that profiles model layers and assigns them to the most efficient compute unit. The scheduler maintains a cost model tracking latency and power per layer type, dynamically routing operations to NPU for efficiency-critical layers and GPU for throughput-critical sections.

Manages server configuration through declarative YAML/JSON files specifying model paths, quantization settings, batch sizes, context windows, and hardware targets. The system supports environment variable substitution, config validation against a schema, and hot-reloading of non-critical settings without server restart.

Provides official Docker images with ROCm, model weights, and Lemonade pre-installed, enabling single-command deployment on AMD GPU-equipped systems. Images include layer caching optimization for fast rebuilds and multi-stage builds to minimize final image size. Docker Compose templates are provided for orchestrating multi-model deployments.

Exposes LLM inference through a standards-compliant HTTP REST API with OpenAI-compatible endpoints, supporting both request-response and server-sent events (SSE) streaming for token-by-token output. The server implements connection pooling, request queuing with configurable concurrency limits, and graceful backpressure handling to prevent memory exhaustion under high load.

Manages multiple LLM checkpoints in a single server process, implementing on-demand model loading into GPU/NPU memory and automatic unloading when models are idle. The system tracks model memory footprints, implements LRU (least-recently-used) eviction policies, and pre-allocates memory pools to minimize allocation latency during model swaps.

Automatically converts full-precision models to lower-bit representations (INT8, INT4, FP8) optimized for target hardware, using calibration data to minimize accuracy loss. The system profiles model layers, selects per-layer quantization strategies (symmetric vs asymmetric, per-channel vs per-tensor), and generates optimized kernels for the chosen precision on AMD GPUs/NPUs.

Automatically groups multiple inference requests into batches to maximize GPU/NPU utilization, implementing a token-level scheduler that pads sequences to common lengths and overlaps computation across requests. The scheduler maintains a priority queue, implements configurable batch size limits and timeout thresholds, and uses continuous batching to avoid blocking on slow requests.

Efficiently manages the key-value cache for transformer models using sliding window attention (only attending to recent tokens) and KV cache compression techniques. The system implements configurable context window sizes, automatic cache eviction policies, and memory-mapped storage for very long contexts, reducing memory overhead from O(n²) to O(n) for long sequences.

Provides fine-grained control over text generation behavior through configurable sampling strategies including temperature scaling, top-k filtering, nucleus (top-p) sampling, and repetition penalties. The server implements efficient GPU-side sampling kernels that apply these constraints in parallel across batch elements, avoiding CPU bottlenecks during token selection.

Accepts models in multiple formats (GGUF, SafeTensors, ONNX, PyTorch) and automatically converts them to an optimized internal representation for AMD hardware. The system detects format, validates model architecture, applies format-specific optimizations (e.g., GGUF quantization patterns, ONNX operator fusion), and maintains a compatibility layer for models trained on different frameworks.

Instruments the inference pipeline to measure latency at multiple granularities: per-request, per-batch, per-layer, and per-operation. The profiler tracks GPU kernel execution time, memory bandwidth utilization, and identifies bottlenecks (memory-bound vs compute-bound layers). Results are exposed via metrics endpoints and logged for offline analysis.