firecrawl-mcp

Exposes Firecrawl's web scraping engine through the Model Context Protocol (MCP), enabling LLM agents to invoke scraping operations as native tools. Routes requests to either Firecrawl's cloud infrastructure or self-hosted instances based on configuration, abstracting transport complexity behind a unified MCP resource interface. Implements request/response marshaling to convert between MCP's JSON-RPC protocol and Firecrawl's REST API contract.

Accepts a URL and optional JSON schema, then uses Firecrawl's backend to fetch the page and extract structured data matching the provided schema. The extraction leverages LLM inference (via Firecrawl's backend) to intelligently map page content to schema fields, handling variations in HTML structure and content layout. Returns validated JSON conforming to the schema, enabling downstream processing without manual parsing.

Tracks API quota usage per request and enforces client-side rate limits to prevent exceeding Firecrawl's quota. Maintains running counters of requests, bytes processed, and API costs. Provides quota status queries and warnings when approaching limits. Implements token bucket or sliding window rate limiting to smooth request distribution.

Supports streaming scraped content incrementally as it becomes available, rather than buffering entire pages in memory. Useful for large pages (10MB+) that would exceed memory limits or cause long latencies if fully buffered. Returns content as a stream of chunks with optional progress callbacks. Enables real-time content processing without waiting for full page completion.

Allows users to define custom extraction rules using CSS selectors, XPath, or regex patterns as fallback when LLM-based schema extraction fails or is unavailable. Supports rule composition (multiple selectors with AND/OR logic) and field mapping. Provides deterministic, fast extraction for well-structured pages without LLM latency.

Accepts an array of URLs and optional scraping parameters, then submits them to Firecrawl's batch processing pipeline. Implements asynchronous job tracking with polling or webhook callbacks, aggregating results as jobs complete. Handles partial failures gracefully, returning per-URL status (success/error) alongside extracted content. Enables efficient processing of 10s-1000s of pages without blocking the MCP client.

Accepts a search query and optional parameters (number of results, search engine, language), then uses Firecrawl's search capability to find URLs and optionally scrape the top results. Combines search index lookup with on-demand scraping, returning both search metadata (title, snippet, URL) and full page content. Enables LLM agents to research topics by searching and immediately extracting relevant information.

Scrapes a URL and returns content formatted as clean, LLM-optimized markdown with preserved structure (headings, lists, tables, code blocks). Removes boilerplate (navigation, ads, footers) and normalizes formatting to maximize token efficiency and readability for language models. Includes optional metadata extraction (title, author, publish date) in YAML frontmatter.

Scrapes URLs that require JavaScript execution by delegating to Firecrawl's headless browser backend (Puppeteer/Playwright). Waits for specified selectors or timeouts to ensure dynamic content is fully loaded before extraction. Supports cookie/session injection for authenticated scraping. Returns fully rendered HTML or extracted content after JavaScript execution completes.

Automatically detects and removes non-content elements (navigation menus, sidebars, ads, footers, cookie banners) from scraped pages using heuristic analysis and optional ML-based content detection. Preserves main article/content body while stripping structural noise. Configurable aggressiveness levels allow tuning between content preservation and noise removal.

Maintains a cache of previously scraped URLs within a configurable TTL (time-to-live), returning cached results for duplicate requests without re-scraping. Implements content-based deduplication to detect semantically identical pages (same content, different URLs). Reduces API quota usage and latency for repeated scraping patterns.

Supports custom HTTP headers, proxy URLs, and user-agent strings to enable scraping from different geographic regions, bypassing IP-based restrictions, and authenticating to protected resources. Passes proxy and header configuration to Firecrawl's backend, which applies them during page fetch. Enables scraping of geo-restricted or authentication-required content.

Implements automatic retry logic for transient failures (timeouts, rate limits, temporary server errors) using exponential backoff with configurable max retries and backoff multiplier. Distinguishes between retryable errors (429, 503) and permanent failures (404, 403), avoiding wasted retries on unrecoverable errors. Returns detailed error information including failure reason, retry count, and final status.