LiteLLM

Provides a single OpenAI-compatible API surface that automatically detects and routes requests to 100+ LLM providers (OpenAI, Anthropic, Google, Azure, Ollama, etc.) without code changes. Uses provider detection logic in get_llm_provider_logic.py that parses model names and environment variables to instantiate the correct provider client, normalizing request/response formats across heterogeneous APIs. Supports streaming, non-streaming, and async completion calls with unified error handling and retry logic.

Distributes requests across multiple LLM provider instances using configurable routing strategies (round-robin, least-busy, cost-optimized, latency-based). The Router class maintains per-provider health metrics, tracks request queues, and implements weighted load distribution based on user-defined priorities. Supports dynamic model deployment where multiple providers can serve the same logical model endpoint, with automatic failover when a provider becomes unavailable or exceeds rate limits.

Provides standalone proxy server (FastAPI-based) that acts as a centralized gateway for all LLM requests, implementing authentication, rate limiting, cost tracking, and observability at the gateway level. Supports pass-through endpoints that forward requests directly to providers without modification, enabling compatibility with existing OpenAI-compatible clients (LangChain, LlamaIndex, etc.). Includes management endpoints for API key management, team management, spend analytics, and health checks. Can be deployed as Docker container, Kubernetes pod, or standalone binary.

Continuously monitors provider health by making periodic test requests to each provider and tracking response latency, error rates, and availability. Maintains per-provider health status (healthy, degraded, unhealthy) and automatically marks providers as unavailable if they fail health checks. Integrates with alerting systems (email, Slack, PagerDuty) to notify operators of provider issues. Provides health check dashboard showing provider status, latency trends, and error patterns.

Implements content safety and guardrails system that validates requests and responses against user-defined rules. Supports built-in guardrails (PII detection, prompt injection detection, toxicity filtering) and custom validators via Python functions or external APIs. Guardrails can be applied to requests (before sending to LLM), responses (after receiving from LLM), or both. Integrates with external safety services (e.g., Perspective API for toxicity) and supports custom guardrail chains where multiple validators are applied sequentially.

Enables logical grouping of models under named access groups (e.g., 'fast-models', 'cheap-models', 'reasoning-models') that can be referenced in API calls without knowing specific model names. Supports wildcard routing where requests to 'gpt-4*' automatically route to the latest GPT-4 variant, and model aliases where 'my-gpt-4' maps to a specific provider's model. Integrates with RBAC to restrict which users can access which model groups. Simplifies model management by decoupling application code from specific model names.

Provides compatibility layer for OpenAI's Assistants API, enabling applications built for OpenAI Assistants to work with other providers (Anthropic, Google, etc.) through LiteLLM. Supports assistant creation, thread management, message history, and file uploads. Implements feature parity where assistants can use tools, retrieval (RAG), and code interpreter across multiple providers. Translates Assistants API calls to provider-specific APIs, handling differences in tool calling, file handling, and state management.

Provides unified interface for reasoning and extended thinking features across providers (OpenAI o1, Anthropic extended thinking, etc.). Automatically detects provider capabilities and enables extended thinking when requested, handling differences in token counting, cost calculation, and response formatting. Supports configurable thinking budgets and thinking display options (show/hide internal reasoning). Integrates with cost tracking to account for higher costs of reasoning models.

Integrates with vector stores (Pinecone, Weaviate, Milvus, etc.) and provides RAG (Retrieval-Augmented Generation) capabilities for semantic search and document retrieval. Supports embedding generation via multiple providers (OpenAI, Cohere, Hugging Face), automatic document chunking and indexing, and semantic search queries. Integrates retrieved documents into LLM context automatically, with configurable retrieval strategies (top-k, similarity threshold, reranking). Supports both synchronous and asynchronous retrieval.

Implements MCP (Model Context Protocol) server gateway that enables LLMs to interact with external tools and services via standardized protocol. Supports MCP clients connecting to LiteLLM proxy, which routes tool calls to registered MCP servers. Implements A2A (Agent-to-Agent) protocol for agent-to-agent communication. Provides tool registry and automatic tool discovery from MCP servers. Integrates with function calling to enable seamless tool use across providers.

Automatically calculates and tracks API costs for every LLM call by parsing response token counts and applying provider-specific pricing models. The cost_calculator.py module maintains a pricing database for 100+ models with per-token input/output rates, and integrates with the proxy's spend tracking system to aggregate costs by user, team, or organization. Supports real-time spend alerts, budget enforcement, and detailed cost analytics exported in FOCUS format for FinOps integration.

Caches LLM responses using both exact-match and semantic similarity strategies to reduce redundant API calls. Exact-match caching stores responses by hashing the complete request (model, messages, parameters), while semantic caching uses embeddings to identify similar prompts and return cached responses for semantically equivalent queries. Integrates with Redis for distributed caching across multiple instances, with configurable TTL and cache invalidation policies. Supports dynamic cache controls via request headers to override caching behavior per-call.

Implements multi-level fallback chains where requests automatically retry on failure using exponential backoff, and fall back to alternative providers if the primary provider fails. Maintains per-provider cooldown timers to prevent hammering a temporarily unavailable provider, and tracks failure patterns to identify systemic issues. Supports configurable retry policies (max attempts, backoff strategy, retriable error codes) and fallback ordering (e.g., try GPT-4, then Claude, then Llama). Integrates with health checks to mark providers as unhealthy and route around them.

Enforces rate limits and quotas at multiple levels: per-user, per-team, per-API-key, and per-provider. Uses token bucket algorithms to smooth traffic and prevent burst overloads, with configurable limits on requests-per-minute, tokens-per-minute, and concurrent requests. Integrates with the proxy's database to persist quota state, and supports dynamic quota adjustment via management APIs. When limits are exceeded, requests are either queued (with configurable wait time) or rejected with appropriate HTTP status codes (429 Too Many Requests).

Provides unified function calling interface that normalizes tool/function definitions across providers (OpenAI, Anthropic, Google, etc.) with automatic schema validation and response parsing. Accepts function schemas in JSON Schema format, translates them to provider-specific formats (OpenAI's tools, Anthropic's tool_use, Google's function_declarations), and parses responses to extract function calls with validated arguments. Supports parallel function calling (multiple functions in single response), automatic retry on validation errors, and integration with external function registries.

Implements provider-native prompt caching (OpenAI, Anthropic, Google) by automatically detecting cacheable content and injecting cache control headers into requests. Supports both prefix caching (cache system prompts and context) and semantic caching (cache based on message similarity). Tracks cache hit rates and cost savings from cached tokens, and provides configuration options to control cache behavior (e.g., min_cache_tokens, cache_creation_tokens). Automatically manages cache lifecycle, including invalidation when prompts change.

Provides multi-tenant architecture with role-based access control (RBAC), API key management, and organization/team/user hierarchy. Each tenant can have multiple teams, each team can have multiple users, and each user can have multiple API keys with granular permissions (e.g., read-only, specific model access). Integrates with SCIM and SSO for enterprise identity management, and supports object-level permissions where users can only access resources they own or are granted access to. API keys are hashed and stored securely, with automatic rotation and expiration policies.

Provides extensible callback system that hooks into request/response lifecycle, enabling custom logging, monitoring, and observability integrations. Built-in callbacks integrate with Langfuse, MLflow, Arize, and other observability platforms, automatically logging request metadata (model, tokens, cost, latency, provider), response data, and errors. Supports message redaction for privacy (e.g., removing PII before logging), custom callbacks for application-specific logging, and structured logging output (JSON) for easy parsing by log aggregation systems.