Outlines

Enforces LLM outputs to conform to arbitrary JSON schemas by integrating with the model's token generation loop. Uses a finite state machine (FSM) built from the schema to mask invalid tokens at each generation step, ensuring 100% schema compliance without post-hoc parsing or validation. Works by computing allowed next tokens based on the current parse state of the JSON being generated.

Constrains LLM token generation to match a regular expression pattern by building a DFA (deterministic finite automaton) from the regex and masking invalid tokens at each step. Enables generation of phone numbers, URLs, dates, or any text matching a specific pattern without post-generation validation or rejection sampling.

Allows developers to hook into the generation loop with custom callbacks that can inspect or modify constraint state, token masks, or sampling behavior. Callbacks are invoked at each generation step, enabling custom logic for constraint relaxation, adaptive masking, or constraint-aware logging. Supports both synchronous and asynchronous callbacks.

Enables combining multiple constraints (e.g., JSON schema AND regex pattern) by computing the intersection of their token masks at each generation step. Supports constraint chaining where the output of one constraint feeds into the next, enabling complex constraint hierarchies. Masks are combined using logical AND to ensure all constraints are satisfied simultaneously.

Integrates with llama.cpp to enable constrained generation on quantized models (GGUF format), allowing efficient inference on CPU or low-VRAM devices. Applies token masking at the llama.cpp C++ level, minimizing Python overhead. Supports all constraint types (JSON, regex, CFG) on quantized models with minimal performance degradation.

Provides a unified interface for constrained generation via OpenAI and Anthropic APIs by translating Outlines constraints into native function-calling schemas. Handles schema conversion, API request formatting, and response parsing automatically. Supports both JSON mode (OpenAI) and tool_use (Anthropic) with transparent fallback and retry logic.

Enforces LLM outputs to conform to a context-free grammar by parsing the generated tokens against the grammar rules and masking tokens that would violate the grammar. Supports arbitrary CFGs (more expressive than regex) for generating code snippets, mathematical expressions, or domain-specific languages. Uses an Earley parser or similar to track valid next tokens based on the current parse state.

Provides a unified Python API for constrained generation across heterogeneous LLM backends (transformers, vLLM, llama.cpp, OpenAI, Anthropic, etc.) by abstracting the token generation interface. Each backend implements a common interface for token sampling and masking, allowing the same constraint code to run on local models, quantized models, or cloud APIs without modification.

Optimizes throughput for constrained generation by batching multiple requests and applying constraints at the batch level using vLLM's paged attention and continuous batching. Masks tokens for all sequences in a batch simultaneously, reducing per-request overhead and enabling higher throughput than sequential generation. Integrates with vLLM's scheduler to maintain constraint compliance across dynamic batches.

Provides a templating system for building prompts that automatically integrate with constraint definitions, allowing developers to define prompts and their expected output schemas in a single configuration. Supports Jinja2-style templating with variable substitution and constraint metadata, enabling reusable prompt-constraint pairs without manual synchronization.

Integrates constraint-based token masking with the model's sampling layer by intercepting logits before sampling and zeroing out invalid tokens. Supports multiple sampling strategies (greedy, temperature-based, top-k, top-p) while maintaining constraint compliance. Masks are computed efficiently using precomputed FSMs or parse states to avoid redundant computation.

Enables token-by-token streaming of constrained outputs, yielding valid tokens as they are generated while maintaining constraint compliance. Maintains constraint state across streamed tokens and updates masks incrementally, allowing real-time output display without buffering the entire response. Supports streaming to HTTP clients, file handles, or custom callbacks.

Accepts Pydantic models as constraint definitions and automatically converts them to JSON schemas for constrained generation. Supports Pydantic v1 and v2 with field validation, nested models, and complex types. Enables type-safe constraint definitions where the schema is derived from Python type annotations.

Caches compiled finite state machines (FSMs) for regex and JSON schema constraints across multiple generation calls, avoiding redundant compilation overhead. Uses memoization keyed by constraint definition (schema, regex, or grammar) to reuse FSMs for identical constraints. Supports in-memory and persistent caching strategies.