marvin

Converts Python functions decorated with @ai markers into AI-executable tasks by parsing docstrings and type hints to build LLM prompts, then executes them against configured LLM backends (OpenAI, Anthropic, etc.). Uses introspection to extract function signatures and constraints, automatically marshaling inputs/outputs between Python types and LLM-compatible formats.

Provides a unified interface to multiple LLM backends (OpenAI, Anthropic, Ollama, local models) through a provider-agnostic client that handles authentication, request formatting, and response parsing. Abstracts away provider-specific API differences so users can swap backends without changing application code.

Enables efficient batch processing of large datasets through AI functions using map-reduce patterns, automatic batching, and parallel execution. Handles chunking of large inputs, concurrent execution across multiple workers, and aggregation of results without requiring manual parallelization code.

Automatically parses LLM responses into typed Python objects (dataclasses, Pydantic models, enums) by embedding JSON schemas in prompts and validating outputs against expected types. Uses LLM-native schema support (OpenAI's JSON mode, Anthropic's structured output) when available, falling back to regex/JSON parsing for other providers.

Executes AI functions asynchronously using Python's asyncio, with built-in support for streaming responses (token-by-token output) and concurrent task execution. Implements async context managers and generators to handle long-running LLM calls without blocking, enabling real-time response streaming to clients.

Enables AI agents to break down complex tasks into subtasks, plan execution sequences, and reason about dependencies using chain-of-thought prompting and tool-use patterns. Agents can call other AI functions, evaluate intermediate results, and adapt plans based on outcomes, implementing a simple form of autonomous task orchestration.

Maintains conversation history and context across multiple AI function calls, automatically managing message buffers and context windows to fit within LLM token limits. Implements sliding-window context management and optional summarization to preserve long-term memory while staying within token budgets.

Provides a templating system for building dynamic prompts with variable substitution, conditional blocks, and formatting helpers. Templates are compiled from Python f-strings or Jinja2-style syntax, allowing prompts to adapt based on runtime context, user input, and task-specific parameters without hardcoding.

Estimates API costs and token usage for LLM calls before execution, tracking actual usage across requests and providing cost breakdowns by model and provider. Uses provider-specific token counting libraries (tiktoken for OpenAI, Claude's token counter) to accurately predict costs without making dummy API calls.

Implements automatic retry mechanisms for failed LLM calls with exponential backoff, jitter, and configurable fallback strategies (e.g., try different provider, use cheaper model). Handles transient errors (rate limits, timeouts) gracefully while distinguishing from permanent failures (invalid input, authentication errors).

Provides first-class integration with popular Python web frameworks through middleware, route decorators, and async context managers. Enables seamless embedding of AI functions into web endpoints with automatic request/response marshaling, streaming support, and error handling.