doctor

Doctor implements a distributed crawling system using crawl4ai for HTML fetching paired with Redis-backed job queuing. The Web Service accepts crawl requests via REST API, enqueues them to Redis, and the Crawl Worker processes jobs asynchronously, enabling non-blocking crawl operations at scale. This microservice architecture decouples request handling from resource-intensive crawling, allowing the system to handle multiple concurrent crawl jobs without blocking client requests.

The Crawl Worker uses langchain_text_splitters to break extracted HTML text into semantically meaningful chunks before embedding. This capability supports multiple splitting strategies (character-based, token-based, recursive) to optimize chunk size for downstream embedding models, ensuring that semantic boundaries are preserved and chunks fit within embedding model token limits. The chunking strategy is configurable per crawl job, allowing optimization for different content types and embedding models.

Doctor uses environment variables and configuration files to control system behavior (embedding provider, Redis connection, DuckDB path, crawl parameters). This configuration-driven approach allows deployment-time customization without code changes, supporting different environments (dev, staging, production) with different settings. Configuration covers embedding model selection, database paths, queue settings, and crawl parameters like timeout and retry logic.

Doctor abstracts embedding generation through litellm, enabling support for multiple embedding providers (OpenAI, Anthropic, local models) without changing core code. The Crawl Worker generates vector embeddings for each text chunk using the configured provider, storing both the chunk text and its vector representation in DuckDB. This abstraction allows switching embedding providers by configuration change, supporting cost optimization and model selection without code modification.

Doctor stores text chunks and their vector embeddings in DuckDB with vector search support (VSS), enabling semantic similarity search across indexed content. The system computes vector similarity between query embeddings and stored chunk embeddings, returning ranked results based on cosine similarity. This capability allows LLM agents to retrieve contextually relevant information from indexed websites using natural language queries, without requiring keyword matching.

Doctor exposes its search and crawl capabilities through the Model Context Protocol (MCP), enabling LLM agents to discover, crawl, and search indexed websites as native tools. The MCP server translates agent tool calls into Doctor API requests, allowing agents to autonomously trigger crawls, search indexed content, and retrieve specific documents. This integration enables LLM agents to extend their knowledge beyond training data by accessing live web content through a standardized protocol.

Doctor exposes a REST API for querying indexed documents, allowing applications to search crawled content and retrieve specific chunks by semantic similarity or metadata filters. The API accepts search queries, executes vector similarity search against the DuckDB index, and returns ranked results with source URLs and chunk content. This capability enables non-agent applications to access indexed web content programmatically.

Doctor provides REST API endpoints for creating, monitoring, and managing crawl jobs with persistent status tracking. Jobs are enqueued to Redis with metadata (URL, status, progress, error messages), and clients can poll job status endpoints to track progress from queued → processing → completed/failed. The system stores job metadata in DuckDB, enabling historical tracking and error diagnosis. This capability allows applications to manage long-running crawl operations and handle failures gracefully.

The Crawl Worker extracts plain text from crawled HTML using content extraction logic that removes boilerplate (navigation, ads, scripts), preserving main content. This extraction happens after crawl4ai fetches the page, converting raw HTML into clean text suitable for chunking and embedding. The extraction strategy balances content preservation with noise removal, ensuring that extracted text is semantically meaningful without excessive markup or irrelevant elements.

Doctor's architecture supports horizontal scaling of crawl workers by adding multiple worker instances that consume jobs from the same Redis queue. Each worker independently processes crawl jobs, extracts text, generates embeddings, and stores results in the shared DuckDB instance. The Redis queue ensures job distribution across workers without duplication, enabling linear scaling of crawl throughput by adding more worker instances.

Doctor uses DuckDB as the primary data store for crawled content, storing raw text, text chunks, vector embeddings, and job metadata in structured tables. The database schema includes tables for jobs (status, metadata), documents (raw content, source URL), and chunks (text, embeddings, chunk index). This persistent storage enables long-term retention of indexed content and supports both semantic search (via VSS) and metadata-based queries (URL filtering, date ranges).