mcp-smart-crawler

Implements the ModelContextProtocol server specification to expose web crawling as a standardized tool interface for AI models and agents. The server registers itself as an MCP resource provider, allowing Claude and other MCP-compatible clients to invoke crawling operations through the protocol's tool-calling mechanism without direct HTTP integration.

Uses Playwright's cross-browser automation engine to crawl dynamic, JavaScript-rendered web content by controlling real browser instances (Chromium, Firefox, WebKit). Handles page navigation, DOM interaction, and content extraction with full JavaScript execution support, enabling crawling of SPAs and AJAX-heavy sites that fail with static HTTP clients.

Enforces configurable timeouts for page navigation, content loading, and JavaScript execution, preventing crawls from hanging indefinitely on slow or unresponsive sites. Implements memory and CPU limits per browser instance, with automatic process termination if limits are exceeded, protecting against resource exhaustion from malicious or poorly-designed pages.

Extracts specific content from crawled pages using CSS selectors or XPath expressions, allowing users to define which DOM elements to extract without parsing entire HTML. The crawler applies selectors to the rendered DOM after JavaScript execution, returning structured data mapped to selector patterns.

Provides specialized crawling logic for Xiaohongshu (Chinese social media platform) content, handling platform-specific authentication, dynamic content loading, and anti-bot measures. Implements custom navigation patterns and wait conditions tailored to XHS's JavaScript-heavy interface and content discovery mechanisms.

Manages browser page navigation with configurable wait conditions (waitUntil: 'load', 'domcontentloaded', 'networkidle'), timeout management, and error handling for failed navigations. Implements retry logic and graceful degradation when pages fail to load, allowing crawls to continue with partial data or fallback strategies.

Manages multiple simultaneous crawl requests from MCP clients by queuing and dispatching them to available Playwright browser instances. Implements request buffering and basic concurrency control to prevent resource exhaustion, though without explicit connection pooling or load balancing across multiple browser processes.

Provides command-line interface for starting the MCP server with configurable options (port, browser type, resource limits). Parses CLI arguments and environment variables to initialize the Playwright browser pool and MCP protocol handler, exposing the crawler as a tool to connected MCP clients.

Manages Playwright browser instance creation, reuse, and cleanup across multiple crawl requests. Implements browser pooling to avoid expensive startup overhead, with automatic cleanup of stale or crashed browser processes and reconnection logic for failed instances.

Implements error handling for common crawling failures (network errors, timeouts, selector mismatches, browser crashes) with graceful degradation strategies. Returns partial results or error details to MCP clients rather than crashing, allowing agents to decide whether to retry, use fallback data, or abandon the crawl.

Allows customization of HTTP request headers (User-Agent, Referer, Accept-Language) to mimic different browsers and devices, with built-in user-agent rotation to avoid detection as a bot. Supports device emulation profiles (mobile, tablet, desktop) with corresponding viewport and user-agent combinations, enabling crawling of mobile-specific content and bypassing simple bot detection.