Lobe Chat

Abstracts 100+ LLM providers (OpenAI, Anthropic, Google, Ollama, Azure, etc.) through a unified provider configuration system that maps provider-specific APIs to a standardized request/response interface. Uses a provider registry pattern with model definitions stored in localized JSON, enabling runtime provider switching without code changes. Supports streaming, non-streaming, and function-calling modes across heterogeneous provider APIs.

Enables chat messages to include images (vision), text-to-speech synthesis, and speech-to-text transcription through a unified message format that stores media as file references or base64 blobs. Vision requests are routed to providers supporting image input (GPT-4V, Claude 3 Vision, Gemini); TTS/STT are handled via provider-specific APIs or third-party services. Message rendering pipeline detects media types and renders appropriate UI components (image previews, audio players, transcript displays).

Provides a multi-stage Dockerfile that builds the Next.js application and creates a minimal production image with Node.js runtime. Docker Compose configuration orchestrates LobeChat service with Postgres database, Redis cache, and optional Qdrant vector database. Supports environment variable injection via .env file, volume mounts for persistent data, and health checks for service monitoring. Image is optimized for size (~500MB) using node:alpine base and dependency pruning.

Implements a localization system using i18next with JSON translation files for 50+ languages (English, Chinese, Spanish, French, German, Japanese, etc.). Language detection is automatic based on browser locale or user preference. Translation keys are organized hierarchically (e.g., 'chat.message.send', 'agent.config.temperature'). Model descriptions and provider names are localized separately to enable dynamic updates without code changes. Client-side language switching is instant without page reload.

Uses Zustand for client-side state management (chat messages, agent configuration, UI state) with a minimal API surface (create store, subscribe, getState). Server state (user data, knowledge bases, agents) is synchronized via React Query (TanStack Query) with automatic caching, invalidation, and background refetching. State updates are optimistic (UI updates immediately) with server confirmation, enabling responsive UX. Persists critical state (user preferences, recent agents) to localStorage for offline access.

Builds the UI using Ant Design component library with custom theming and responsive breakpoints. Layout is mobile-first, adapting from mobile (320px) to desktop (1920px) using CSS media queries and Ant Design's Grid system. Supports dark mode with theme switching (light/dark/auto). Components are optimized for touch on mobile (larger tap targets, swipe gestures) and keyboard navigation for accessibility. Responsive design is tested across browsers and devices.

Implements a user memory system that stores persistent user preferences (language, theme, default model), conversation summaries, and long-term context across sessions. Memory is stored in a dedicated database table with automatic summarization of old conversations using LLM. Memory is injected into agent context as a system message, enabling agents to personalize responses based on user history. Supports memory editing and deletion for privacy control.

Integrates Redis for caching frequently accessed data: LLM responses, embedding vectors, knowledge base search results, and user sessions. Cache keys are generated from request parameters (model, prompt, user_id) with configurable TTL (time-to-live). Cache invalidation is triggered by data mutations (agent config changes, knowledge base updates). Supports cache warming for popular queries. Cache hit/miss metrics are tracked for performance monitoring.

Provides a plugin marketplace and runtime that executes tools via two mechanisms: (1) native function-calling APIs (OpenAI, Anthropic, Google) with schema-based function definitions, and (2) Model Context Protocol (MCP) servers for standardized tool execution. Plugins are registered in a marketplace with metadata (name, description, icon, permissions), and the agent runtime selects and invokes plugins based on LLM-generated function calls. Tool execution is sandboxed with timeout and resource limits.

Implements a complete RAG (Retrieval-Augmented Generation) system: users upload documents (PDF, TXT, Markdown), which are chunked using configurable strategies (fixed-size, semantic, recursive), embedded using provider-specific embeddings (OpenAI, Ollama), stored in a vector database (Postgres with pgvector, or Qdrant), and retrieved via semantic search. Retrieved chunks are injected into the LLM context window with relevance scoring. Supports knowledge base organization by topic/collection with per-KB access controls.

Implements a multi-tenant architecture with user authentication (OAuth, API keys, local accounts), role-based access control (RBAC) with roles like admin, user, guest, and fine-grained permissions (create agents, access knowledge bases, manage plugins). Each user has isolated sessions, chat history, and knowledge bases stored in a relational database (Postgres). Session tokens are validated server-side; user context is injected into all requests via middleware. Supports team/organization-level access controls for shared resources.

Allows users to define AI agents with custom system prompts, model selection, temperature/top-p settings, and tool/knowledge base bindings. Agents are stored as configuration objects with metadata (name, description, avatar, tags). At runtime, the agent configuration is loaded, the system prompt is prepended to the message history, and the agent runtime routes requests to the selected LLM provider with the configured parameters. Supports agent memory (short-term conversation history, long-term user memory) that persists across sessions.

Stores chat messages in a relational database (Postgres) with full-text search and topic-based organization. Messages are linked to sessions (conversations) and topics (subtopics within a session), enabling hierarchical conversation organization. Supports message editing, deletion, and pinning. Message content is stored as JSON (role, content, metadata), enabling rich formatting and media references. Retrieval is optimized with indexes on user_id, session_id, and created_at for fast pagination and filtering.

Handles file uploads (images, documents, audio) with client-side validation and server-side processing. Files are stored in S3 (or local file system) with metadata (filename, size, MIME type, upload timestamp) tracked in database. Supports automatic file type detection and processing: PDFs are converted to text/images, images are resized and optimized, audio files are transcribed. File references are stored as URLs or base64 blobs in messages, enabling efficient retrieval and rendering.

Implements streaming chat responses using Server-Sent Events (SSE) for HTTP and WebSocket for bidirectional communication. LLM responses are streamed token-by-token to the client, enabling real-time display of AI-generated text. Supports streaming for function calls, tool execution results, and multi-turn conversations. Client-side streaming parser reconstructs complete responses from chunks and handles connection errors with automatic reconnection.

Provides a Vercel deployment template with pre-configured environment variables and build settings. Deployment is triggered via a 'Deploy to Vercel' button that forks the repository and configures serverless functions for API routes. Environment variables (LLM API keys, database credentials, S3 bucket) are set via Vercel dashboard or .env file. Build process uses Next.js static generation and serverless functions, optimizing for Vercel's infrastructure. Supports automatic deployments on git push with CI/CD integration.