Robust LLM extractor for websites in TypeScript

Extracts structured data from website HTML by leveraging LLM reasoning to understand semantic content and convert unstructured markup into typed JSON schemas. Uses prompt engineering and schema validation to guide LLM output toward consistent, machine-readable formats without requiring manual parsing rules or CSS selectors.

Validates LLM-extracted data against a provided JSON schema and automatically coerces types (string to number, date parsing, enum matching) to ensure output conforms to expected structure. Implements schema validation logic that catches hallucinations or malformed LLM responses before returning to user code.

Abstracts differences between LLM providers (OpenAI, Anthropic, Ollama, etc.) behind a unified interface, allowing users to swap providers or use multiple models without changing extraction logic. Handles provider-specific API differences, token counting, and model-specific prompt formatting transparently.

Processes multiple URLs or HTML documents in parallel with configurable concurrency limits, managing rate limits and API quota to avoid throttling. Implements queue-based batching with retry logic, allowing extraction of hundreds of pages without manual rate-limit handling or request throttling.

Automatically constructs and optimizes prompts for LLM extraction by injecting schema definitions, examples, and HTML context in a structured format. Implements prompt templates that guide the LLM toward consistent extraction behavior and reduce hallucination through few-shot examples and explicit instructions.

Implements intelligent fallback mechanisms when extraction fails, including retry with different models, simplified schema extraction, or manual review workflows. Detects extraction failures (schema validation errors, LLM refusals, timeouts) and applies recovery strategies without user intervention.

Cleans and normalizes HTML before LLM extraction by removing noise (scripts, styles, ads, tracking), extracting main content, and normalizing whitespace and encoding. Uses heuristics or DOM analysis to identify and preserve semantically important content while reducing token usage and improving extraction accuracy.

Caches extraction results by URL or content hash to avoid redundant LLM calls for identical or previously-extracted content. Implements configurable cache backends (in-memory, Redis, file-based) and deduplication logic to detect when the same content has been extracted before.

Tracks extraction quality metrics (success rate, schema compliance, confidence scores, latency) and provides observability into extraction pipeline behavior. Emits structured logs and metrics that integrate with monitoring systems to detect extraction degradation or anomalies.

Provides pre-built extraction templates and adapters for common websites (e-commerce, news, social media) that optimize prompts, schemas, and preprocessing for known website patterns. Allows users to select a template instead of defining extraction logic from scratch, with customization options for site-specific variations.