just-every/mcp-read-website-fastMCP Server27/100 via “configurable concurrent worker-based web fetching with polite crawling”
** - Fast, token-efficient web content extraction that converts websites to clean Markdown. Features Mozilla Readability, smart caching, polite crawling with robots.txt support, and concurrent fetching with minimal dependencies.
Unique: Combines configurable worker pools with robots.txt compliance and User-Agent spoofing prevention in a single fetching layer, rather than treating crawling politeness as a separate concern, ensuring ethical behavior is enforced at the network boundary
vs others: More ethical and sustainable than naive concurrent scrapers because robots.txt compliance and rate limiting are built-in rather than optional, reducing risk of IP blocks and legal issues when crawling third-party content at scale