khoj

Indexes user documents (markdown, PDFs, web pages) into PostgreSQL with vector embeddings, enabling semantic search via cosine similarity matching. Uses a content processing pipeline that extracts, chunks, and embeds documents through configurable embedding models, then retrieves contextually relevant passages to augment chat responses. The search engine supports multiple content sources (local files, web URLs, Obsidian vaults) with unified indexing through database adapters.

Routes chat requests through a provider-agnostic conversation pipeline that supports OpenAI (GPT), Anthropic (Claude), Google Gemini, and local LLMs (Llama, Qwen, Mistral via Ollama/LlamaCPP). The chat processor retrieves relevant context from the semantic search index, constructs a system prompt with retrieved passages, and streams responses back to clients. Implements conversation history management via Django ORM with per-user conversation threads and message persistence.

Provides an Obsidian plugin that indexes the user's vault into Khoj's knowledge base and enables semantic search within Obsidian. The plugin watches for file changes and incrementally updates the index, supporting live synchronization of new notes. Implements bidirectional integration: users can search their vault from Khoj chat, and Khoj can suggest related notes from the vault. The plugin uses Obsidian's API for file access and the Khoj backend API for indexing and search.

Provides an Emacs plugin that enables inline chat and search within Emacs buffers. Users can select text, ask Khoj questions about it, and receive responses inline. The plugin supports semantic search of indexed documents and integrates with Emacs' completion and buffer management systems. Implements streaming response rendering in Emacs buffers with syntax highlighting for code blocks.

Provides Docker and Docker Compose configurations for self-hosted deployment of the full Khoj stack (backend, PostgreSQL, frontend). Includes environment-based configuration management through .env files and Django settings, supporting customization of LLM providers, embedding models, search engines, and other services. The deployment supports both development (docker-compose.yml) and production (prod.Dockerfile) configurations with Gunicorn WSGI server for production.

Implements a content processing pipeline with pluggable extractors for different file types (PDF, markdown, HTML, plain text, Obsidian). Each extractor converts the source format to normalized text, which is then chunked and embedded. The pipeline supports custom extractors through a plugin interface, allowing users to add support for new file types. Chunking strategies are configurable (fixed size, semantic, sliding window) with metadata preservation (source, timestamp, section).

Implements streaming response delivery through both HTTP Server-Sent Events (SSE) and WebSocket protocols, enabling real-time response rendering on clients. The streaming processor chunks LLM responses and sends them incrementally, reducing perceived latency and enabling progressive rendering. Supports streaming for chat responses, search results, and agent execution logs. Clients can subscribe to response streams and render content as it arrives.

Implements an agent system that decomposes user requests into subtasks, selects appropriate tools (web search, code execution, image generation, MCP servers), and executes them in sequence with result aggregation. The agent uses the LLM to reason about tool selection via function-calling APIs (OpenAI, Anthropic native support) or prompt-based tool selection for other providers. Tool execution is sandboxed through subprocess isolation for code execution and API-based execution for external tools, with results fed back into the agent loop for iterative refinement.

Provides a specialized research workflow that iteratively searches the web, retrieves results, synthesizes findings, and generates follow-up queries based on gaps in knowledge. The research mode uses the agent system to orchestrate multiple web searches with semantic deduplication of results, then aggregates findings into a structured research report. Implements a loop that continues searching until confidence threshold is met or iteration limit reached, with each iteration refining the search query based on previous results.

Integrates image generation capabilities through OpenAI DALL-E, Hugging Face Stable Diffusion, and local image generation models. The image processor accepts natural language prompts from chat or agent tasks, generates images through the selected provider, and returns URLs or base64-encoded images. Supports diagram generation through specialized prompts that guide the LLM to create structured image descriptions suitable for visualization tools.

Executes Python code snippets in a sandboxed subprocess environment with output capture and error handling. The code executor accepts code strings from the agent or chat, runs them with restricted permissions, captures stdout/stderr, and returns results to the agent loop. Implements timeout protection (default 30 seconds) and resource limits to prevent runaway execution. Results are streamed back to clients for real-time feedback.

Provides multiple client interfaces (Next.js web app, Emacs plugin, Obsidian plugin, desktop/mobile apps) that all connect to a unified FastAPI backend through REST APIs. Each client implements its own UI/UX while sharing the same backend services (chat, search, agents, settings). The backend exposes REST endpoints for all operations, with WebSocket support for streaming responses. Authentication is handled centrally through the backend with token-based auth (JWT) and multi-method support (password, OAuth).

Integrates with MCP (Model Context Protocol) servers to extend the agent's tool capabilities beyond built-in tools (web search, code execution, image generation). The MCP processor discovers available tools from registered MCP servers, converts them to function-calling schemas compatible with LLM providers, and executes them through the agent loop. Supports both local MCP servers and remote endpoints with automatic schema translation and error handling.

Manages conversation threads and message history through Django ORM models (Conversation, Message) stored in PostgreSQL. Each user has isolated conversation threads with full message history, metadata (timestamps, token counts, model used), and optional titles. The conversation manager supports retrieving conversation context for augmentation, archiving old conversations, and exporting conversation history. Implements efficient context window management by truncating older messages when approaching token limits.

Implements authentication through multiple methods: password-based login, OAuth (Google, GitHub), and API key authentication. Uses JWT tokens for session management with configurable expiration. Authorization is role-based (user, admin) with per-user resource isolation (conversations, settings, indexed documents). The authentication backend (UserAuthenticationBackend) integrates with Django ORM for user management and supports both web clients (cookie-based) and API clients (token-based).