SurfSense

SurfSense implements a pluggable connector architecture supporting 28+ data sources (Google Drive, Slack, Notion, GitHub, Jira, etc.) through a standardized OAuth integration flow and periodic indexing pipeline. Each connector implements a common interface for authentication, document fetching, and metadata extraction, with background task processing handling continuous synchronization without blocking the main application. The system abstracts away source-specific API complexity through a unified document ingestion pipeline that normalizes heterogeneous data formats into a common internal representation.

SurfSense combines vector similarity search (semantic embeddings) with BM25 full-text search and applies a reranking step to produce hybrid results that balance semantic relevance with keyword matching. The system stores document chunks as embeddings in a vector database and maintains full-text indices for keyword-based retrieval, then merges results using a configurable scoring strategy. This hybrid approach enables finding documents that match both conceptual meaning and specific terminology, critical for research and knowledge work where both types of relevance matter.

SurfSense provides multiple deployment options including Docker containerization for quick setup and manual installation for custom environments. The system includes database migrations (Alembic), environment configuration templates, and comprehensive documentation for both deployment methods. This enables organizations to self-host SurfSense on their infrastructure, maintaining full control over data, security, and customization without relying on cloud services or third-party hosting.

SurfSense implements internationalization (i18n) infrastructure in the frontend application, supporting multiple languages through a translation system. The system includes language selection in the UI, translated strings for all user-facing text, and support for right-to-left languages. This enables teams in different regions to use SurfSense in their native language without requiring separate deployments or code modifications.

SurfSense implements a document mention system that tracks which documents are referenced in conversations, enabling users to see which knowledge base items are actively used in discussions. When users mention documents in chat or when the RAG system retrieves documents, the system records these references with timestamps and context. This creates a knowledge graph showing relationships between conversations and documents, enabling discovery of related discussions and understanding of document usage patterns.

SurfSense implements a retrieval-augmented generation (RAG) pipeline where user queries trigger hybrid search to retrieve relevant document chunks, which are then passed as context to an LLM for response generation. The system tracks source attribution throughout the pipeline—maintaining references from retrieved chunks back to original documents—and surfaces citations in the chat interface. The chat architecture supports multi-turn conversations with thread management, allowing users to ask follow-up questions while maintaining context and citation lineage across the conversation.

SurfSense abstracts LLM provider selection through a configuration layer that allows different roles (admin, user) to select from 100+ supported models across multiple providers (OpenAI, Anthropic, Ollama, local models, etc.). The system maintains provider-specific configurations (API keys, model parameters, rate limits) and routes requests to the appropriate provider based on user role and workspace settings. This abstraction enables organizations to enforce cost controls (e.g., cheaper models for certain users), support multiple LLM providers simultaneously, and switch providers without code changes.

SurfSense implements multi-tenancy through SearchSpaces—isolated workspaces where teams can manage documents, conversations, and LLM configurations independently. Each SearchSpace has its own document index, conversation history, and member list, with role-based access control (RBAC) determining what actions each user can perform (view documents, create conversations, manage connectors, etc.). The system maintains workspace isolation at the database level, ensuring data from one SearchSpace cannot leak to another, while supporting team membership management with invitations and role assignments.

SurfSense implements a document processing pipeline that ingests raw documents, chunks them into semantically meaningful segments (respecting document structure), extracts metadata (title, author, source URL), and generates embeddings for each chunk. The system preserves metadata throughout the pipeline, maintaining links from chunks back to source documents and original content locations. This enables citation tracking, source attribution in search results, and reconstruction of document context when displaying search results.

SurfSense includes a podcast generation capability that transforms chat conversations or document collections into structured podcast scripts with multiple speakers, dialogue, and narrative flow. The system uses LLMs to synthesize information from source materials, generate speaker personas, create dialogue between speakers, and produce audio-ready scripts. This enables teams to convert research findings or internal knowledge into consumable audio content without manual scripting or production work.

SurfSense provides a Chrome/Firefox browser extension that enables users to capture web content, highlight text, and search the knowledge base directly from any webpage. The extension communicates with the SurfSense backend through API calls, allowing users to add web pages to their knowledge base, search existing documents, and access chat conversations without leaving their browser. This extends SurfSense functionality into the user's browsing workflow, enabling seamless knowledge capture and retrieval.

SurfSense surfaces the LLM's reasoning process by capturing and displaying 'thinking steps'—intermediate reasoning, document retrieval steps, and decision-making logic—alongside final chat responses. This transparency feature helps users understand how the AI arrived at its answer, which documents influenced the response, and where the reasoning might be uncertain. The system integrates thinking steps with citation tracking, showing users both the reasoning process and the source documents that informed each step.

SurfSense implements a thread-based conversation system where each conversation maintains its own context, message history, and document references. The system preserves conversation state across sessions, allowing users to return to previous conversations and continue discussions with full context. Threads support branching (creating new conversations from existing ones) and archiving, enabling users to organize conversations by topic or project. The architecture maintains message ordering, timestamps, and metadata for each turn, enabling conversation replay and audit trails.