WebDataSource

Discovers follow-up pages from a seed URL by applying CSS or XPath selectors to extract links, then automatically queues those pages as Download Tasks for crawling. Uses a job-based architecture where each crawl operation references a Job Config and returns a set of async tasks that can be polled for completion status, enabling multi-level hierarchical crawls across site structures.

Extracts text content and HTML attributes from DOM elements matching CSS or XPath selectors, returning structured JSON with specified field names. Works on already-downloaded pages (via Download Tasks) and supports multi-field extraction in a single operation, enabling conversion of unstructured HTML into agent-consumable JSON documents.

Retrieves metadata and status information about jobs via GetJobsInfo and GetJobConfig tools, allowing agents to list active/completed jobs, inspect job configurations, and track job history. Provides visibility into job state without requiring agents to maintain separate state tracking, enabling job management and monitoring workflows.

Executes complex multi-level crawl-then-scrape workflows defined in MDR (multi-level data retrieval) configs, where each level can crawl pages, apply selectors to discover follow-up URLs, and extract structured data. Uses cursor-based pagination to return results in batches via GetCrawlMdrData, enabling agents to process large result sets incrementally without loading entire datasets into memory.

Polls the status of asynchronous crawl/scrape operations (Download Tasks) to check completion, retrieve error details, and inspect request/response metadata. Returns status objects containing task state, HTTP error codes, network errors, and selector match failures, enabling agents to implement retry logic and error handling without direct access to underlying HTTP details.

Retrieves relevant documents from previously indexed web resources using semantic similarity search, taking a natural language query and returning ranked documents. Implements retrieval-augmented generation (RAG) pattern by maintaining an index of crawled/scraped content and matching incoming queries against that index, enabling agents to answer questions grounded in web data without re-crawling.

Creates, updates, and retrieves job configurations that define crawl/scrape parameters (seed URLs, selectors, extraction rules, pagination settings). Stores configurations persistently in WDS, allowing agents to reuse, modify, and restart jobs without redefining parameters. Supports both simple job creation (StartJob) and complex hierarchical plans (CrawlMdrConfig), with helper tools for building and updating configs.

Provides an optimized job startup path (StartJobForFastWebResourceEvaluation) designed for rapid assessment of web resources without full crawling overhead. Intended for scenarios where agents need to quickly evaluate whether a resource is relevant or accessible before committing to full crawl/scrape operations, reducing latency and resource consumption for initial resource discovery.

Allows agents to continue or resume evaluation of web resources across multiple invocations via UpdateJobToContinueWebResourceEvaluation, enabling stateful evaluation workflows where agents can pause, inspect intermediate results, and decide whether to continue deeper evaluation. Maintains job state across calls, allowing agents to implement adaptive crawling strategies that adjust based on discovered content.

Exposes WebDataSource crawling/scraping capabilities as an MCP (Model Context Protocol) server via HTTP transport, allowing integration with VS Code and other MCP-compatible IDEs. The server runs as part of the WDS API Server (default localhost:2807/mcp) and handles tool invocation, resource management, and async task tracking through standard MCP protocol, eliminating the need for custom API client code.

Retrieves large result sets from completed crawl/scrape operations in batches using cursor-based pagination via GetCrawlMdrData. Agents provide a cursor token (initially null) and receive a batch of JSON documents plus a next cursor token, enabling incremental processing of large datasets without loading entire results into memory. Supports streaming-like consumption patterns for memory-efficient data processing.