llm (Simon Willison)

Implements a dual sync/async base class hierarchy (Model, AsyncModel, KeyModel, AsyncKeyModel) defined in llm/models.py that abstracts away provider-specific details. Any model—whether OpenAI, Anthropic, local, or plugin-provided—inherits from these base classes and implements prompt() and execute() methods, allowing identical code to work across all providers without conditional logic or provider detection.

Automatically logs all model interactions to a local SQLite database (logs.db) with full conversation state, including prompts, responses, model metadata, tokens used, and timestamps. The Conversation class in llm/models.py maintains multi-turn dialogue state and can be serialized/deserialized from the database, enabling conversation resumption, audit trails, and historical analysis without external services.

Exposes a Python library interface (llm module) that allows developers to interact with models programmatically without using the CLI. Core functions like llm.get_model(), model.prompt(), and model.execute() provide a simple API for single-turn and multi-turn interactions. The API supports both sync and async patterns, enabling integration into web frameworks, scripts, and applications. Responses are returned as Response objects with methods for accessing text, JSON, and usage statistics.

Supports generating embeddings for large batches of text via the embed_batch() method on EmbeddingModel, which is more efficient than calling embed() repeatedly. The system tracks token usage and can estimate costs based on model pricing. Batch operations are optimized to minimize API calls and reduce costs, particularly useful for processing large document corpora.

Provides methods to query model capabilities at runtime, such as whether a model supports function calling, vision, streaming, or structured output. The Model base class exposes properties and methods that describe supported features, enabling applications to adapt behavior based on model capabilities without hardcoding provider-specific logic. This enables graceful degradation when features are unavailable.

Implements a plugin system using Python entry points (setuptools) that auto-discovers and registers custom models, tools, and templates at runtime. The plugin manager in llm/cli.py scans installed packages for llm.models, llm.tools, and llm.templates entry points, dynamically loading them without modifying core code. Plugins can extend functionality by subclassing Model, Tool, or Template base classes.

Enables models to invoke Python functions by defining a Tool class with a function() decorator and optional JSON schema. The Toolbox class collects related tools and prepares them for model consumption via prepare() method, which generates tool schemas compatible with OpenAI and Anthropic function-calling APIs. When a model invokes a tool, llm executes the corresponding Python function and returns the result to the model, enabling multi-step reasoning and external action.

Supports constrained generation where models must return JSON matching a provided schema. The Prompt class accepts a schema parameter, and the Response class provides a json() method that parses and validates the model output against the schema. Some providers (e.g., OpenAI with JSON mode) enforce this at the API level; others validate client-side. This enables reliable extraction of structured data (e.g., entities, classifications) from unstructured model outputs.

Supports attaching images, audio, files, and other media to prompts via the Prompt class's attachments parameter. The Fragments system encapsulates different media types and their metadata, allowing models to process multi-modal inputs. Attachments are serialized and logged to the SQLite database, enabling conversation history to preserve media references. Different models support different attachment types (e.g., OpenAI supports images, Anthropic supports images and PDFs).

Provides an EmbeddingModel base class for generating vector embeddings from text. The embedding system stores vectors in a separate SQLite database (embeddings.db) with associated metadata, enabling semantic search and similarity operations. Plugins can provide embedding models (e.g., OpenAI's text-embedding-3-small, local models via Ollama). The embed() and embed_batch() methods support both single and bulk embedding generation.

Provides an interactive chat interface via the llm chat command that maintains conversation state, handles multi-turn interactions, and streams model responses to the terminal in real-time. The CLI uses the Conversation class to manage history and the AsyncModel interface for non-blocking streaming. Responses are displayed incrementally as tokens arrive, improving perceived latency and enabling early interruption.

Supports defining reusable prompt templates with variable placeholders that can be instantiated with different values. Templates are stored as files or registered via the plugin system, and can include system prompts, tools, and schemas. The template system uses simple string substitution (e.g., {variable_name}) to inject values at runtime, enabling prompt reuse across different contexts without code duplication.

Allows users to define aliases for models (e.g., 'default' -> 'gpt-4-turbo') and configure model parameters (temperature, max_tokens, system prompts) via configuration files in ~/.llm/. The configuration system supports per-model settings and global defaults, enabling users to customize model behavior without modifying code. Aliases are stored in a configuration file and loaded at CLI startup.