Diffbot

Automatically extracts structured data from arbitrary web pages without requiring manual rule definition or CSS selectors. Uses computer vision combined with NLP to detect and classify page elements (articles, products, organizations, discussions, events) and convert them into clean, normalized JSON output. The system learns visual patterns across diverse page layouts to identify relevant fields without configuration.

Crawls websites by discovering and following links across configurable URL scopes (50 to 50,000+ URLs per crawl), then automatically applies the Extract API to each discovered page to build structured datasets. Operates asynchronously, allowing batch processing of entire site hierarchies without manual URL enumeration. Supports configurable crawl depth, scope limits, and automatic link discovery.

Processes unstructured text (1-10,000 characters per document) to automatically identify and extract named entities (people, organizations, locations, etc.), infer relationships between them, and perform topic-level sentiment analysis. Uses NLP models to parse text without requiring pre-defined entity schemas or training data, returning structured entity and relationship records.

Queries a pre-indexed knowledge graph containing 10+ billion entities (246M+ organizations, 1.6B+ articles, 3M+ products, 23k+ events, and people records) to retrieve structured entity records with 50+ fields for organizations (categories, revenue, locations, investments, etc.) and 20+ fields for products (brand, images, reviews, offers, prices). Enables fast entity resolution and relationship mapping without crawling or extraction.

Enriches existing person and organization datasets by automatically fetching and extracting web-sourced attributes (company revenue, employee count, locations, funding, leadership, product information, etc.) and merging them into provided records. Uses web crawling and extraction to supplement incomplete or outdated records with current information from public sources.

Integrates Diffbot's extraction and enrichment capabilities into non-technical platforms (Excel, Google Sheets, Zapier, Tableau) via custom connectors and query interfaces. Enables business users to extract web data, enrich records, and visualize results without writing code — Excel and Sheets use visual query builders or Diffbot Query Language (DQL), while Zapier enables trigger-based enrichment workflows and Tableau enables dashboard integration.

Offers optional datacenter proxy routing for Extract and Crawl API requests to rotate IP addresses and avoid rate limiting or IP-based blocking by target websites. Requests routed through Diffbot's proxy infrastructure appear to originate from different IPs, enabling crawling of sites with aggressive rate limiting or IP-based access controls. Costs 2 credits per page (vs. 1 credit without proxy).

Operates on a credit-based consumption model where each API operation (Extract, Natural Language, Knowledge Graph export) consumes a fixed number of credits, with monthly credit allotments varying by subscription tier (Free: 10k/month, Startup: 250k/month, Plus: 1M/month, Enterprise: custom). Rate limits vary by tier (Free: 5 calls/min, Startup: 5 calls/sec, Plus: 25 calls/sec), and overage charges apply pro-rata at the plan's per-credit rate after monthly allotment is exhausted.