Configurable Memory Eviction Policies

via “lru cache-based model eviction with multi-backend resource management”

OpenAI-compatible local AI server — LLMs, images, speech, embeddings, no GPU required.

Unique: Implements LRU eviction at the application layer (ModelLoader) rather than relying on OS-level memory management, providing explicit control over which models stay resident and enabling predictable memory behavior across heterogeneous backends. The eviction policy coordinates across all active backends, ensuring system-wide memory constraints are respected.

vs others: Unlike vLLM (which requires sufficient VRAM for all models) or Ollama (which loads one model at a time), LocalAI's LRU eviction enables running multiple models simultaneously on constrained hardware by intelligently swapping models based on access patterns.