airweave

Airweave implements a source connector architecture that abstracts heterogeneous data sources (Google Docs, Linear, Intercom, Trello, etc.) through a unified interface. Each connector implements OAuth integration via an Auth Provider System, handles incremental sync using cursor-based tracking to avoid re-processing, and manages token refresh lifecycle. The Temporal Workflow System orchestrates sync jobs with configurable schedules (one-time, recurring, continuous), while the Entity Processing Pipeline streams entities through a queue with backpressure handling and concurrency controls to prevent source API throttling.

Airweave implements a Search System built on Vespa for distributed vector similarity search across indexed entities. The search pipeline accepts natural language queries, converts them to embeddings, and retrieves candidates using Vespa's ranking framework. The Agentic Search capability allows AI agents to refine queries iteratively — agents can inspect initial results, reformulate queries, and re-rank results based on relevance signals. The search operations pipeline supports hybrid search (combining vector similarity with BM25 keyword matching) and filters by collection, source, and metadata breadcrumbs to scope results to relevant document hierarchies.

Airweave provides a web-based Dashboard with React frontend (state management via Zustand) for managing collections, viewing sync status, and monitoring usage. The Collection Management UI enables creating/editing collections and managing source connections. The dashboard displays sync progress (entities processed, errors, duration) and allows triggering manual syncs. Real-Time Updates and SSE enable live progress updates without polling. The Usage Limits and Billing UI shows API usage, sync counts, and billing status. The Application Structure and Routing uses React Router for navigation between dashboard sections. OAuth Callback Flow is handled transparently in the UI for source connection setup.

Airweave supports self-hosted deployment via Docker containers. The Docker and Deployment documentation provides Dockerfiles for backend, frontend, and worker services. Configuration Management via environment variables and YAML files (dev.integrations.yaml, prd.integrations.yaml, self-hosted.integrations.yaml) enables customization of OAuth providers, storage backends, and feature flags. The backend service uses PostgreSQL for relational data and Qdrant for vector storage; both can be self-hosted or cloud-managed. The start.sh script automates local setup with Docker Compose. Self-hosted deployments have full control over data residency and can customize integrations (e.g., add custom OAuth providers).

Airweave implements Incremental Sync and Cursors to avoid re-processing all entities on every sync. Source connectors track a cursor (e.g., last_modified_timestamp, page_token) that marks the point of the last successful sync. On subsequent syncs, the connector fetches only entities modified after the cursor, reducing API calls and processing time. The Sync System stores cursors in PostgreSQL and updates them after each successful sync. Change detection is source-specific: some sources provide modification timestamps, others use pagination tokens. The Entity Processing Pipeline processes only new/changed entities, making incremental syncs much faster than full syncs.

Airweave uses a Qdrant Multi-Tenant Architecture where each organization's vectors are isolated in separate Qdrant collections, with metadata stored in PostgreSQL. The QdrantDestination API implements a write path that batches entity embeddings and writes them to Qdrant with error handling and retry logic. PostgreSQL stores the relational schema (collections, source connections, sync metadata) and serves as the source of truth for entity relationships and breadcrumbs. The dual-write pattern ensures consistency: vectors in Qdrant are indexed for search, while PostgreSQL maintains referential integrity and enables complex queries (e.g., 'find all entities from source X synced after timestamp Y').

Airweave exposes search capabilities as a Model Context Protocol (MCP) server, allowing Claude and other MCP-compatible agents to invoke search as a native tool. The MCP Server Architecture defines a search tool schema that agents can call with natural language queries and filters. The MCP Search Tool handles query parsing, invokes the underlying Search System (Vespa-backed), and returns results in a format agents can reason about. This enables agents to autonomously search the knowledge base without explicit function-calling code — the agent sees search as a first-class capability in its tool registry.

Airweave provides a Connect Widget — an embeddable React component that handles the full OAuth flow for connecting sources. The Connect Widget Architecture manages OAuth Callback Flow internally: it initiates OAuth with the source platform, handles the redirect callback, exchanges the authorization code for tokens, and stores credentials securely. The Connect Client SDKs (JavaScript/TypeScript) expose a simple API for embedding the widget in external applications. Connect Session Management tracks widget state (pending, authenticated, error) and enables parent applications to listen for connection events. This eliminates the need for applications to implement OAuth flows themselves.

Airweave organizes indexed entities into Collections, which are logical groupings of related data (e.g., 'Q1 2024 Research', 'Customer Support Docs'). Collections can contain entities from multiple sources, and each entity maintains breadcrumb metadata (source, document_id, parent_id) that preserves document hierarchy. The Collections API enables CRUD operations on collections and supports filtering search results by collection. Breadcrumbs enable hierarchical queries (e.g., 'find all entities under parent document X') and preserve context for agents (e.g., 'this result came from a Linear ticket in the Q1 Planning project'). This enables agents to reason about result provenance and scope searches to relevant document trees.

Airweave manages the full lifecycle of source connections: OAuth authentication, token storage in PostgreSQL, automatic token refresh before expiry, and error handling with retry logic. The Source Connection Lifecycle pattern tracks connection state (authenticated, expired, error) and implements Token Management and Refresh that automatically refreshes OAuth tokens before they expire, preventing sync failures. The Factory Pattern and Context Building construct source-specific clients with refreshed credentials at sync time. Error Handling and Resilience implements exponential backoff and dead-letter queues for failed syncs, enabling operators to retry failed connections without manual intervention.

Airweave uses Temporal Workflows to orchestrate data syncs as reliable, resumable jobs. The Temporal Worker Architecture runs activities (source-specific sync logic) within workflow contexts that handle retries, timeouts, and state persistence. Workflows define sync schedules (one-time, recurring via cron, continuous polling) and manage the full sync lifecycle: entity fetching, processing, and writing to storage. The Sync Orchestration layer coordinates multiple sources syncing in parallel while respecting rate limits and backpressure. Progress Tracking and Metrics capture sync progress (entities_processed, errors, duration) and enable operators to monitor sync health via dashboards. Schedule Management allows dynamic schedule updates without restarting workers.

The Entity Processing Pipeline implements stream-based processing of entities from sources through a queue with backpressure handling. Entities are streamed from source connectors into an in-memory queue, processed in batches (normalization, embedding generation), and written to storage. The Source Stream and Queue Management layer implements backpressure: if the queue fills up, source fetching pauses until downstream processing catches up. Concurrency and Backpressure controls limit parallel processing to prevent overwhelming source APIs or downstream services (embedding models, vector stores). This enables high-throughput syncs without resource exhaustion or API throttling.

Airweave exposes a comprehensive REST API (documented via OpenAPI/Fern) for programmatic management of collections, sources, and search. The Collections API enables CRUD operations on collections and membership. The Source Connections API manages OAuth connections and sync state. The Sources API lists available source types and their configuration schemas. The Search API accepts queries and returns ranked results. The API uses standard REST conventions (GET, POST, PUT, DELETE) and returns JSON responses. Authentication is via API keys stored in PostgreSQL. The API enables external applications to integrate Airweave without using the web UI or SDKs.