Fetch MCP Server

Implements MCP tool registration that exposes HTTP GET/POST fetching as a callable tool through the Model Context Protocol's JSON-RPC transport layer. The server registers a 'fetch' tool with input schema validation, handles HTTP requests via Python's urllib or requests library, and returns structured responses that conform to MCP tool result primitives, enabling LLM clients to invoke web fetching as a first-class capability without direct HTTP knowledge.

Transforms fetched HTML content into Markdown format optimized for LLM processing using a conversion library (likely html2text or similar). The server parses HTML structure, preserves semantic meaning (headings, lists, links, emphasis), strips unnecessary styling and scripts, and outputs clean Markdown that reduces token consumption and improves LLM comprehension compared to raw HTML. This conversion happens server-side before returning results to the MCP client.

Implements robots.txt parsing and compliance validation before fetching URLs, checking the User-Agent against disallowed paths and crawl-delay directives defined in the target domain's robots.txt file. The server fetches and caches robots.txt entries, evaluates requested URLs against allow/disallow rules, and either permits or blocks the fetch based on compliance. This ensures the MCP server respects web scraping conventions and legal/ethical boundaries without requiring clients to implement their own robots.txt logic.

Defines the 'fetch' tool's input schema using JSON Schema format (with required fields like 'url' and optional fields like 'method', 'headers', 'body') and validates incoming MCP tool call requests against this schema before processing. The server uses the MCP SDK's tool registration mechanism to declare the schema, and the framework automatically validates inputs, returning structured validation errors if the request doesn't match the schema. This ensures type safety and prevents malformed requests from reaching the HTTP fetching logic.

Manages the MCP server's startup, shutdown, and transport initialization using the Python SDK's Server class and async context managers. The server initializes the MCP protocol handler, registers tools (fetch, etc.) during startup, establishes stdio or network transport for client communication, and gracefully shuts down resources on exit. This lifecycle management ensures the server is ready to receive MCP requests and properly cleans up connections when the client disconnects or the server terminates.

Catches HTTP errors (4xx, 5xx, network timeouts, connection failures) and maps them to structured MCP error responses with descriptive messages. The server distinguishes between client errors (404 Not Found, 403 Forbidden), server errors (500 Internal Server Error), and network errors (timeout, DNS failure), returning appropriate error codes and messages that clients can interpret. This ensures fetch failures are communicated clearly without crashing the server or leaving the MCP connection in an inconsistent state.

Allows clients to specify custom HTTP headers (including User-Agent, Authorization, Accept, etc.) in the fetch tool request, enabling access to APIs that require specific headers or authentication. The server passes these headers through to the HTTP request, allowing clients to override the default User-Agent (which might be blocked by some sites) or add authentication tokens. This flexibility enables the fetch tool to work with a wider range of web services and APIs without requiring server-side configuration changes.

Implements a maximum response body size limit (typically 1-10 MB) to prevent memory exhaustion from fetching extremely large files or responses. When a response exceeds the limit, the server truncates the body and returns a truncation indicator, allowing clients to know that the full content was not retrieved. This protects the server from out-of-memory errors and ensures fetch operations complete in reasonable time, though it may lose information from large documents.

Implements the fetch server as an MCP server that communicates with clients through the MCP protocol using a transport layer (stdio, SSE, or custom). The server uses the Python MCP SDK to handle protocol-level concerns (JSON-RPC message framing, request/response routing, error serialization) while the application code focuses on tool logic. The transport layer is abstracted away, allowing the same server code to work with different transport mechanisms without modification.