Scrapling

Implements a three-tier fetcher system (Fetcher → BrowserFetcher → StealthyFetcher) where each level adds capabilities while maintaining identical Response object contracts. All fetchers return Response objects that inherit from Selector, enabling developers to write parsing code once and switch fetching strategies without refactoring. Uses lazy imports via __getattr__ to defer loading heavy dependencies (Playwright, browser engines) until first access, reducing initial import overhead.

Automatically relocates DOM elements when page structure changes during interaction, using fallback selector strategies (CSS → XPath → text content matching) to recover element references after JavaScript mutations. Implements element caching with invalidation detection to identify when selectors no longer match their original targets, then attempts recovery using alternative selector types or proximity-based matching. This enables robust scraping of single-page applications where DOM structure shifts during user interactions.

Response factory and converter system enables custom type handlers that transform raw HTML into structured Python objects (dataclasses, Pydantic models, TypedDicts). Converters can be registered per-response-type, enabling automatic deserialization of HTML into domain-specific types. Supports chaining converters for multi-step transformations (HTML → intermediate dict → final dataclass). Integrates with Spider framework's Item system for declarative data extraction pipelines.

Browser configuration system (BrowserConfig) manages Playwright browser lifecycle, context creation, and tab pooling. Supports headless/headed mode, viewport configuration, device emulation, and custom launch arguments. Tab pooling within a single browser context reduces memory overhead compared to per-request browser spawning. Implements resource cleanup with context managers and automatic tab reuse across requests. Supports browser-specific features like geolocation spoofing, timezone configuration, and locale emulation for testing localized content.

Command-line interface and interactive shell enable exploratory scraping without writing code. CLI supports single-request scraping with selector extraction (scrapling fetch URL --selector 'div.item'). Interactive shell provides REPL-like environment where users can iteratively test selectors, refine queries, and inspect responses. Shell maintains session state across commands, enabling multi-step workflows (fetch → inspect → extract). Supports command history, tab completion, and pretty-printing of HTML and extracted data.

StealthyFetcher layer applies multiple anti-bot detection evasion techniques including user-agent randomization, header spoofing, WebDriver property masking, and behavioral mimicry (random delays, mouse movements, viewport variations). Uses Playwright's stealth plugin architecture to inject JavaScript that masks automation indicators (navigator.webdriver, chrome.runtime detection) and simulates human-like interaction patterns. Integrates with proxy rotation to distribute requests across IP addresses, making detection by rate-limiting or IP-based blocking more difficult.

Response objects inherit from Selector class, providing chainable CSS and XPath query methods that work identically across all fetcher types. Selectors return lists of elements that can be further queried, enabling fluent API patterns like response.css('div.item').xpath('.//span[@class="price"]').text(). Supports both string selectors and compiled selector objects for performance optimization. Parsing is lazy-evaluated; selectors are not executed until .text(), .attr(), or .html() is called, reducing memory overhead for large documents.

Sessions (Session, AsyncSession, BrowserSession) manage connection reuse and browser lifecycle, with browser sessions supporting tab pooling to optimize resource usage. Sessions maintain cookies, headers, and authentication state across multiple requests, enabling workflows that require login or multi-step interactions. Browser sessions pool Playwright tabs within a single browser context, reducing memory overhead compared to spawning separate browser instances. Sessions support proxy assignment per-request or per-session, with automatic rotation strategies.

Spider framework provides a declarative pattern for defining crawl workflows using start_urls, parse() callbacks, and request/response pipelines. Spiders inherit from BaseSpider and define parse() methods that yield Request objects for follow-up crawls or Item objects for data extraction. The framework handles request queuing, deduplication, and response routing automatically. Supports middleware-style request/response processors that can modify requests before sending or transform responses before parsing, enabling cross-cutting concerns like rate-limiting, error handling, and data validation.

Implements configurable wait strategies (WaitForSelector, WaitForNavigation, WaitForFunction, WaitForTimeout) that pause execution until specific page load conditions are met. Strategies can be combined (e.g., wait for selector AND navigation) to handle complex loading patterns. Uses Playwright's built-in wait mechanisms (page.wait_for_selector, page.wait_for_navigation) under the hood, with timeout configuration and exception handling. Enables scraping of content that loads asynchronously after initial page render, such as infinite-scroll feeds or lazy-loaded images.

Proxy management system supports per-request and per-session proxy assignment with automatic rotation strategies (round-robin, random, weighted). Proxies can be configured as simple URL strings or as ProxyConfig objects with authentication credentials. Rotation happens transparently at the fetcher level without application code changes. Integrates with both static HTTP fetchers (via httpx proxy support) and browser fetchers (via Playwright proxy configuration). Supports proxy health checking and automatic fallback to direct connection if proxy fails.

Scrapling exposes web scraping capabilities as an MCP (Model Context Protocol) server, enabling AI agents and LLMs to invoke scraping operations through standardized tool-calling interfaces. The MCP server wraps Fetcher, BrowserFetcher, and StealthyFetcher as callable tools with schema-based function signatures, allowing Claude, GPT, and other LLM-based agents to request web scraping without direct API knowledge. Supports tool parameters for URL, selector, wait conditions, and proxy configuration, with response serialization to JSON for LLM consumption.

AsyncFetcher and AsyncSession provide async/await interfaces for concurrent request execution using Python's asyncio. Supports concurrent HTTP requests via httpx.AsyncClient and concurrent browser operations via Playwright's async APIs. Enables batching of multiple requests with configurable concurrency limits to avoid overwhelming target servers or exhausting local resources. Integrates with Spider framework for concurrent crawl execution with automatic request queuing and deduplication across async tasks.