just-every/mcp-read-website-fast

Extracts clean, semantically meaningful article content from web pages using Mozilla's Readability algorithm, which performs DOM tree analysis to identify and isolate main content while removing boilerplate, navigation, and sidebar elements. The extraction pipeline preserves semantic HTML structure (headings, lists, emphasis) that feeds into downstream Markdown conversion, enabling token-efficient representation for LLM consumption.

Converts extracted semantic HTML into clean, LLM-optimized Markdown using Turndown library with GitHub Flavored Markdown (GFM) plugin, preserving structural elements (headings, lists, code blocks, tables, emphasis) while stripping unnecessary HTML attributes and inline styles. The conversion pipeline maintains link references and code block syntax highlighting hints for downstream processing.

Implements the entire system as a Node.js ES Module package with no native C++ bindings or platform-specific code, enabling seamless deployment across Windows, macOS, and Linux without compilation or platform-specific builds. The pure JavaScript implementation ensures consistent behavior across platforms and simplifies installation and deployment.

Implements a local file-system cache using SHA-256 hashing of URLs as cache keys, storing extracted Markdown with configurable time-to-live (TTL) to avoid redundant fetches and processing. The caching layer sits between the fetch and extraction pipeline, checking cache validity before invoking network requests, reducing latency and bandwidth consumption for repeated URL accesses.

Implements concurrent HTTP fetching using configurable worker pools (default behavior inferred from architecture) to parallelize requests while respecting robots.txt directives and implementing polite crawling practices (rate limiting, User-Agent headers, request delays). The fetching layer manages connection pooling and error handling to enable scalable batch processing without overwhelming target servers or triggering IP blocks.

Extracts all hyperlinks from the original HTML content and preserves them in the Markdown output using reference-style link syntax, enabling knowledge graph construction and cross-document navigation. The extraction pipeline maintains link text, href attributes, and relative URL resolution to ensure links remain valid in downstream processing.

Implements a bootstrap entry point (bin/mcp-read-website.js) that dynamically routes to either CLI or MCP server interfaces based on command arguments, while both interfaces share the same underlying content extraction pipeline (fetchMarkdown.ts). This architecture enables code reuse and consistent behavior across interfaces while allowing each interface to optimize for its specific use case (CLI for scripting, MCP for AI assistant integration).

Implements a Model Context Protocol (MCP) server using stdio transport that exposes web content extraction as a callable tool for AI assistants (Claude, VS Code, Cursor, JetBrains IDEs). The MCP server implements the standard MCP protocol for tool discovery, request/response handling, and error reporting, enabling seamless integration into AI agent workflows without custom client code.

Provides a command-line interface that accepts URL arguments and outputs extracted Markdown to stdout, enabling integration into shell scripts, CI/CD pipelines, and batch processing workflows. The CLI interface supports standard Unix conventions (exit codes, stderr for errors, stdout for results) and can be chained with other command-line tools using pipes and redirection.

Implements the entire system using only 4 runtime dependencies (Mozilla Readability, Turndown, GFM plugin, and HTTP client), avoiding heavy frameworks (Express, Puppeteer, Cheerio) that would increase startup latency and memory consumption. The lean dependency strategy prioritizes fast startup times and low resource overhead critical for AI agent integration where latency impacts user experience.

Produces Markdown output specifically optimized for LLM consumption by removing unnecessary whitespace, using reference-style links to reduce token count, and preserving semantic structure (headings, lists, code blocks) that models understand well. The output format balances readability with token efficiency, enabling longer documents to fit within context windows while maintaining semantic meaning.