Guidance

Guidance uses an immutable Abstract Syntax Tree (AST) of GrammarNode subclasses (LiteralNode, RegexNode, SelectNode, JsonNode, RuleNode, RepeatNode) to define hard constraints on LLM output. The framework compiles these grammar nodes into token-level constraints that are enforced during generation, preventing invalid outputs at the token level rather than post-processing. This works by integrating with the model's tokenizer to ensure only valid token sequences can be generated, achieving 100% constraint satisfaction.

The @guidance decorator transforms Python functions into programs that seamlessly interleave imperative control flow (conditionals, loops, variable assignment) with constrained LLM generation. The framework maintains a stateful execution context (lm object) that accumulates generated text and captured variables, allowing subsequent control flow decisions to depend on LLM outputs. This enables dynamic prompt construction where the next generation step is determined by previous outputs, all within a single continuous execution flow.

Guidance provides visualization tools (Jupyter widgets, HTML output) that display execution traces, showing the sequence of generation steps, constraints applied, and captured variables. The framework logs detailed execution information including token sequences, grammar node traversals, and model state at each step. This enables developers to inspect and debug guidance programs by visualizing how constraints were applied and what the model generated at each stage.

Guidance provides RepeatNode AST nodes and convenience functions (one_or_more, zero_or_more, optional) that enable repetition constraints on generation. These allow developers to specify that a pattern should appear one or more times, zero or more times, or optionally once. The framework compiles these into token-level constraints that enforce the repetition logic during generation, useful for generating lists, repeated structures, or optional elements.

Guidance allows developers to define custom grammar rules using the @guidance decorator, enabling recursive and reusable pattern definitions. Rules can reference other rules, creating complex grammars that are compiled into RuleNode AST nodes. This enables developers to build domain-specific languages (DSLs) and complex output formats by composing simple rules, with the framework handling the compilation and constraint enforcement.

Guidance enables capturing and extracting specific parts of generated text into variables using the capture() function or implicit capture in grammar nodes. Captured variables are stored in the lm state object and can be accessed in subsequent control flow or generation steps. This allows developers to extract structured information from LLM outputs (e.g., entity names, values, decisions) and use them in downstream logic without manual parsing.

Guidance implements token healing by processing text at the character/byte level rather than the token level, ensuring correct tokenization at text boundaries. When constraints are applied or text is concatenated, the framework re-tokenizes affected regions to prevent token boundary misalignment (e.g., a space character being merged into an adjacent token). This is handled by the TokenParser and ByteParser engines, which work with the model's tokenizer to ensure seamless transitions between constrained and unconstrained generation.

Guidance provides a unified model interface that abstracts over multiple backend implementations (LlamaCpp for local inference, Transformers for HuggingFace models, OpenAI/Azure/VertexAI for remote APIs). The framework defines a common Model base class with consistent methods (generate, __call__) that work identically across backends, allowing users to write guidance programs once and execute them on any supported model. Backend selection is transparent to the user; the same @guidance decorated function works with local or remote models by simply changing the model parameter.

Guidance provides a json() function that accepts a JSON schema and generates valid JSON objects that conform to the schema. The framework uses JsonNode AST nodes to compile schemas into token-level constraints, ensuring generated JSON is syntactically valid and matches the schema structure (required fields, types, nested objects). This works by converting JSON schema constraints into grammar rules that are enforced during generation, preventing invalid JSON from being produced.

Guidance provides a select() function that constrains generation to a predefined set of options (strings, numbers, or complex objects). The framework uses SelectNode AST nodes to compile choice constraints into token-level rules, ensuring the model can only generate one of the specified options. This is useful for classification tasks, enum selection, or multi-choice scenarios where the output must be one of a fixed set of values.

Guidance provides a gen(regex=...) function that constrains generation to match a regular expression pattern. The framework uses RegexNode AST nodes to compile regex patterns into token-level constraints via DFA/NFA compilation, ensuring generated text matches the pattern. This enables precise control over output format (e.g., phone numbers, dates, code snippets) without post-processing validation.

Guidance provides chat role abstractions (system, user, assistant) and template functions that simplify multi-turn conversation construction. The framework includes built-in templates for common chat formats (OpenAI, Anthropic, Llama) that automatically handle role markers, message formatting, and token boundaries. Users can define custom chat templates for specific model instruction formats, enabling consistent prompt construction across different models.

Guidance enables tool calling by constraining generation to select from a set of available tools/functions and generate valid function call arguments. The framework uses schema-based constraints to ensure generated function calls match the tool's signature, including parameter types and required fields. This works by combining SelectNode (for tool selection) with JsonNode (for argument validation), creating a unified system for tool use without requiring model-specific function calling APIs.

Guidance supports two execution modes: stateful (default, maintains context across calls) and stateless (cache=True, reuses cached computations). The framework caches intermediate generation results and model states, allowing subsequent calls with similar prefixes to reuse cached tokens rather than regenerating them. This is particularly useful for batch processing or repeated prompts with different suffixes, reducing token consumption and latency.