mcp-smart-crawler

Exposes web crawling capabilities through the Model Context Protocol (MCP) server interface, using Playwright as the underlying browser automation engine. The tool launches a headless browser instance, navigates to URLs, and extracts rendered DOM content, making it accessible to AI models and agents via standardized MCP tool calls rather than direct API integration.

Uses Playwright's headless browser engine to fully render JavaScript-heavy websites and extract the resulting DOM as text or structured data. Unlike static HTTP clients, this waits for page load events, executes client-side JavaScript, and captures the final rendered state, enabling crawling of single-page applications and dynamically-loaded content.

Implements the Model Context Protocol server specification, registering web crawling operations as callable tools with JSON schema definitions. The server exposes tool_list and tool_call handlers that parse incoming MCP requests, validate arguments against schemas, invoke Playwright crawl operations, and return results in MCP-compliant format for consumption by AI models.

Provides selector-based extraction to target specific DOM elements rather than crawling entire pages. Accepts CSS selectors or XPath expressions, uses Playwright's locator API to find matching elements, and extracts their text content, attributes, or inner HTML. This enables precise data extraction from known page structures without parsing full page content.

Manages crawling workflows that span multiple pages, handling browser context persistence, navigation between URLs, and state management across requests. The tool maintains a single Playwright browser instance across multiple crawl operations, allowing efficient reuse of browser resources and enabling workflows like following pagination links or navigating through site hierarchies.

Includes specialized crawling logic for Xiaohongshu (XHS), a Chinese social commerce platform, handling platform-specific HTML structures, dynamic content loading, and anti-bot protections. The tool detects XHS URLs and applies custom extraction rules optimized for feed posts, product listings, and user profiles on that platform.

Runs as a standalone Node.js process that implements the MCP server protocol, handling stdio-based communication with MCP clients (Claude desktop, custom hosts). The tool manages process lifecycle, argument parsing, and server initialization, allowing it to be invoked as a command-line tool that automatically starts the MCP server and waits for client connections.

Implements automatic retry mechanisms for transient failures (network timeouts, temporary 5xx errors, page load failures) with exponential backoff. The tool catches Playwright errors, network errors, and timeout exceptions, retries with increasing delays, and returns structured error information if all retries fail, allowing graceful degradation in crawl workflows.