recursive-llm-ts

Processes arbitrarily large documents and conversations by recursively chunking input into manageable segments, processing each chunk through an LLM, and then recursively combining results until a final output is produced. This enables context windows to effectively exceed the underlying model's token limits by treating the problem as a tree-reduction task where intermediate summaries feed into higher-level processing stages.

Enforces structured output from LLM responses using Zod schemas as the contract layer. The system validates LLM outputs against the schema, automatically retrying with schema-aware prompting if validation fails, and returns fully typed TypeScript objects. This ensures type safety and eliminates JSON parsing errors by making the schema the source of truth for both prompting and validation.

Automatically chunks input text based on the target model's context window size, with configurable overlap between chunks to preserve cross-boundary context. The system calculates token counts accurately, respects semantic boundaries (paragraphs, sentences), and minimizes information loss at chunk edges.

Provides a unified TypeScript interface for multiple LLM providers (OpenAI, Anthropic, and compatible APIs) with automatic provider selection, fallback handling, and streaming response support. The abstraction layer normalizes differences in API signatures, token counting, and response formats, allowing code to switch providers without refactoring.

Abstracts file storage operations (upload, download, delete) across S3 and MinIO backends with a unified TypeScript interface. The system handles multipart uploads for large files, automatic retry with exponential backoff, and configurable storage backends, enabling seamless switching between cloud and self-hosted storage without code changes.

Caches LLM responses based on content hashing of inputs, enabling automatic cache hits for semantically identical requests without explicit cache key management. The system stores cached responses in configurable backends (in-memory, Redis, or file-based) and validates cache freshness using content hashes, reducing redundant API calls and costs.

Implements resilient retry strategies with exponential backoff and jitter for transient failures in LLM API calls and file operations. The system configures retry behavior per operation type (e.g., rate limits vs. network errors), tracks retry attempts, and provides detailed failure telemetry for debugging.

Integrates OpenTelemetry for distributed tracing, metrics collection, and structured logging across LLM calls, file operations, and recursive processing stages. The system automatically instruments key operations, exports traces to compatible backends (Jaeger, Datadog, etc.), and provides detailed performance metrics for optimization.

Processes multiple LLM requests concurrently with configurable concurrency limits, automatic rate limiting, and batch result aggregation. The system manages worker pools, handles partial failures gracefully, and provides progress tracking for long-running batches.

Handles streaming LLM responses with automatic backpressure management, allowing consumers to control the rate of token consumption. The system buffers tokens intelligently, implements flow control to prevent memory overflow, and provides hooks for processing tokens as they arrive.

Validates LLM outputs at each stage of recursive processing against Zod schemas, and feeds validation errors back into the prompt for the next recursion level. This ensures that intermediate outputs conform to expected structures and guides the LLM toward valid final outputs through iterative refinement.