Constrained Decoding With Grammar Based Token Filtering

via “grammar-constrained text generation with token healing”

Microsoft's language for efficient LLM control flow.

Unique: Implements token healing at the text level (not token level) with an immutable GrammarNode AST architecture, allowing constraints to be composed and reused across programs while maintaining correct behavior at token boundaries. The TokenParser/ByteParser dual-engine design handles both token-level and byte-level constraints without requiring external validation passes.

vs others: More efficient than post-generation validation (no retry loops) and more flexible than simple prompt engineering because constraints are enforced during generation, not after, reducing wasted tokens and guaranteeing format compliance on first attempt.

via “constrained decoding with grammar-based token filtering”

C/C++ LLM inference — GGUF quantization, GPU offloading, foundation for local AI tools.

Unique: Implements grammar-based token filtering using finite state machines, ensuring output strictly conforms to GBNF grammars — most inference engines don't support constrained decoding

vs others: Guarantees valid structured output without post-processing, unlike vLLM or Ollama which require validation after generation

via “token-level constraint enforcement with llguidance integration”

A guidance language for controlling large language models.

Unique: Compiles grammar constraints into a state machine that filters token logits during inference, implemented through llguidance C++ extension for performance. This is the core mechanism that enables reliable constraint enforcement without post-processing.

vs others: More reliable than post-processing validation because constraints are enforced during generation, and more efficient than rejection sampling because invalid tokens are filtered rather than sampled and discarded.