gpt-researcher

Routes research tasks across 25+ LLM providers (OpenAI, Anthropic, Ollama, local models, etc.) using a three-tier fallback strategy: primary model for planning, secondary for execution, tertiary for fallback. Implements provider-agnostic abstraction layer that normalizes API differences, handles rate limiting, and manages context windows per model. Supports both cloud and local model deployment without code changes.

Automatically breaks down complex research queries into 5-10 focused sub-queries using the planner agent, then executes them in parallel across multiple concurrent tasks. Each sub-query is independently researched with its own context retrieval and source validation, then results are merged and deduplicated. Uses tree-based query planning to identify dependencies and optimize execution order.

Exposes GPT Researcher as an MCP server, allowing Claude and other MCP-compatible clients to invoke research capabilities as tools. Implements MCP protocol with resource and tool definitions for research queries, configuration, and report retrieval. Clients can call research as a native tool within their workflows. Supports streaming responses for long-running research. Enables integration with Claude projects and other MCP-aware applications without custom API wrappers.

Provides flexible configuration system supporting environment variables, YAML/JSON config files, and programmatic Config class. Centralizes all settings: LLM providers, retrievers, report modes, domain filters, vector stores, etc. Implements configuration validation and defaults. Supports per-environment configurations (dev, staging, production) via config file selection. Environment variables override file-based configs. Enables easy switching between configurations without code changes.

Implements domain-based filtering allowing researchers to include/exclude specific domains from research. Supports whitelist mode (only specified domains) and blacklist mode (exclude specified domains). Validates sources against domain rules before inclusion in reports. Provides built-in domain categories (academic, news, government, etc.) for quick filtering. Enables custom domain rules per research query. Includes domain credibility scoring based on historical performance.

Exports completed research reports in multiple formats: markdown (with inline citations), PDF (formatted with images and styling), and JSON (structured data with metadata). Markdown export preserves source links and citations. PDF export includes table of contents, page numbers, and embedded images. JSON export provides structured access to report sections, sources, and metadata. Supports custom export templates for branded PDF output. Implements format-specific optimizations (e.g., markdown for version control, PDF for sharing).

Maintains research history across sessions, storing completed research queries, reports, and metadata. Implements session management with unique session IDs for tracking research progress. Supports state persistence to database or file system. Enables users to retrieve previous research, compare reports, and build on prior work. Implements automatic cleanup of old sessions. Provides search and filtering across research history. Supports export of research history for audit trails.

Generates research reports in three configurable modes: Standard (quick overview with 3-5 sources), Detailed (comprehensive analysis with 10-15 sources and citations), and Deep (exhaustive research with 20+ sources, fact-checking, and multi-agent review). Each mode uses different prompt templates, source count targets, and validation strategies. Deep mode triggers multi-agent workflow with ChiefEditorAgent orchestrating specialized agents for research, review, and revision.

Retrieves research data from multiple sources: web search (Google, Bing, DuckDuckGo), web scraping with browser automation (Playwright/Selenium), document loading (PDF, DOCX, TXT), and cloud storage (S3, Google Drive). Implements source validation, domain filtering, and deduplication. Each retriever is pluggable; custom retrievers can be added by implementing a standard interface. Handles JavaScript-heavy sites via headless browser execution.

Manages research context across multiple sources using a context manager skill that compresses and prioritizes information to fit within LLM context windows. Implements sliding window compression, source ranking by relevance, and automatic truncation strategies. Tracks token usage per model and adjusts compression aggressively for smaller context windows (e.g., Ollama local models). Deduplicates overlapping information across sources before compression.

Orchestrates specialized research agents (ResearcherAgent, WriterAgent, ReviewerAgent, CuratorAgent) through a ChiefEditorAgent that assigns tasks, manages state, and coordinates review-revision workflows. Each agent has specific skills: ResearcherAgent gathers sources, WriterAgent synthesizes reports, ReviewerAgent validates facts, CuratorAgent filters sources. Implements task dependency tracking, state persistence, and inter-agent communication via message passing. Supports both sequential and parallel agent execution patterns.

Validates and ranks research sources using a CuratorAgent that implements relevance scoring, source credibility assessment, and duplicate detection. Scores sources based on domain authority, content relevance to query, recency, and citation count. Filters out low-quality sources, spam, and duplicates before inclusion in reports. Implements domain-specific credibility rules (e.g., academic sources ranked higher for scientific queries). Provides source metadata including relevance scores and validation reasons.

Exposes research capabilities via FastAPI backend with WebSocket support for real-time streaming of research progress. Clients connect via WebSocket and receive live updates as research progresses: query decomposition, sub-query execution, source retrieval, and report generation. Implements message-based protocol with event types (query_decomposed, sources_found, report_section_generated, etc.). Supports concurrent research sessions with state isolation. Includes REST API for batch research and configuration management.

Automatically generates relevant images for research reports using DALL-E 3 integration. Analyzes report sections and generates descriptive prompts for images that illustrate key concepts. Embeds generated images into markdown reports with captions. Supports image caching to avoid regenerating identical images. Implements fallback to stock image APIs if DALL-E fails. Configurable per-section image generation (e.g., only generate for introduction and conclusion).

Integrates with vector databases (Pinecone, Weaviate, Chroma, Milvus) for semantic search and retrieval-augmented generation. Embeds research sources using sentence transformers or OpenAI embeddings, stores in vector DB, and retrieves semantically similar documents for context. Supports hybrid search combining vector similarity with keyword matching. Implements embedding caching to avoid recomputing embeddings for identical sources. Enables long-term knowledge accumulation across research sessions.