@kb-labs/llm-router

Routes requests across multiple LLM models organized into performance tiers (e.g., fast/cheap vs. slow/capable), selecting the appropriate tier based on request complexity or user-defined routing rules. Implements a decision tree that evaluates incoming prompts against tier criteria and selects the lowest-cost model capable of handling the request, reducing API spend while maintaining quality thresholds.

Automatically cascades requests to alternative models when the primary model fails, times out, or returns an error. Maintains a fallback chain (e.g., GPT-4 → Claude → Llama) and transparently retries with the next model in sequence without requiring application-level retry logic, with configurable backoff and circuit-breaker patterns.

Routes requests to models based on attached metadata (e.g., user tier, request priority, domain) rather than just request content. Evaluates metadata against routing rules at request time to select the optimal model, enabling use cases like 'premium users get GPT-4, free users get GPT-3.5' or 'code generation requests use specialized models'. Metadata can be attached by middleware or application logic before routing.

Provides a single API surface for interacting with multiple LLM providers (OpenAI, Anthropic, Ollama, etc.) by normalizing their different request/response formats into a common schema. Handles provider-specific quirks (token limits, parameter names, response structures) transparently, allowing applications to switch providers without code changes. Implements adapter pattern with provider-specific implementations for each API.

Monitors model availability in real-time by tracking request success/failure rates and response times, automatically removing models from rotation when they exceed error thresholds or timeout consistently. Implements circuit breaker pattern that temporarily disables failing models and periodically tests them for recovery, preventing cascading failures and wasted API calls to unavailable endpoints.

Groups multiple requests destined for the same model and sends them in batch operations where supported (e.g., OpenAI Batch API), reducing per-request overhead and API costs. Tracks costs per model and aggregates them for billing/analytics, providing visibility into which models are consuming budget. Implements batching with configurable window sizes and timeout thresholds to balance latency vs. cost savings.

Automatically optimizes prompts before sending to models by truncating context, removing redundant information, or reformatting based on model token limits and capabilities. Tracks token usage per request and model, enforcing hard limits to prevent exceeding context windows. Implements strategies like sliding window context, summarization, or hierarchical chunking to fit large contexts into model limits while preserving semantic meaning.

Collects latency, throughput, and quality metrics for each model in the routing configuration, enabling data-driven decisions about tier assignments and fallback ordering. Provides built-in benchmarking tools to compare models on representative workloads, with support for custom evaluation metrics. Stores historical performance data to identify trends and detect performance regressions.