How Constrained Decoding Forces Valid JSON: XGrammar, Outlines & GBNF

"JSON mode" is not a prompt trick

Most guides tell you to turn on JSON mode, pass a schema, and move on. That works — but it hides the interesting part. When a provider *guarantees* the output parses, it is not asking the model nicely and retrying on failure. It is doing constrained decoding: at every single token step, it computes which tokens could still lead to valid JSON and forbids all the others. Invalid JSON becomes literally unreachable, not merely unlikely.

If you have read LLM JSON mode & structured outputs for the *how to use it*, this is the *how it works* — and knowing the mechanism is what lets you predict when it will fail.

The one-sentence version

A language model produces a probability (a logit) for every token in its vocabulary at each step. Constrained decoding inserts one operation between the logits and sampling: it sets the logit of every token that would break the grammar to negative infinity.

raw logits ──▶ [ grammar mask: illegal tokens ➜ -inf ] ──▶ softmax ──▶ sample

Because the mask is applied *during* generation rather than checked *after*, there is no parse error to catch, no retry loop, and no fallback parser. The model can only emit a token that keeps the partial output valid.

From JSON Schema to a token mask

The pipeline that turns your schema into that mask has three stages:

1. Schema → grammar. Your JSON Schema is compiled into a formal grammar — an FSM (finite-state machine), a regular expression, or a context-free grammar (CFG). This encodes "a valid document is an opening brace, then a key from this set, then a colon, then a value of this type, …".
2. Grammar → automaton. The grammar becomes an automaton whose state tracks "where are we in the document right now" — inside a string, expecting a comma, three levels deep in nested objects.
3. Automaton → per-step mask. At each decoding step the automaton reports which vocabulary tokens are legal *from the current state*. Everything else is masked.

The subtlety is that a model's tokens are not characters — one token might span a quote and part of a word. So the engine cannot reason character-by-character; it must know, for every token in a 100k+ vocabulary, whether appending it keeps the document parseable. Doing that fast at every step is the entire engineering problem.

FSM vs CFG: why nesting is the dividing line

There are two families of engines, and the difference is exactly the shape of JSON they can express.

• Regex / finite-state engines (Outlines). Outlines pioneered compiling a schema to a regular expression and precomputing, per FSM state, the set of allowed tokens. This is extremely fast for flat structures — a fixed set of fields with scalar values. But a regular language cannot count arbitrary nesting, so deeply recursive JSON is awkward for a pure FSM.
• Context-free / pushdown engines (XGrammar, llama.cpp GBNF). Real JSON Schema needs recursion, so these engines use a CFG with a stack (a pushdown automaton). The stack is what remembers "I have opened three objects and must eventually close all three." This handles arbitrary nesting correctly, at the cost of a runtime stack check.

In 2026 the practical default flipped to CFG engines. XGrammar is now the built-in backend for vLLM, SGLang, TensorRT-LLM, and MLC-LLM, and most stacks auto-select between XGrammar and Outlines depending on whether your constraint needs recursion.

Why XGrammar made the overhead disappear

The old objection to constrained decoding was speed: checking a huge vocabulary against a grammar at every step sounds expensive. XGrammar's insight is that most tokens don't depend on the current parser state.

• It splits the vocabulary into context-independent tokens — ones whose legality is the same regardless of position in the document — which it checks once and caches.
• Only a small set of context-dependent tokens (structural characters like the braces, comma, and quote) must be re-checked at runtime against the stack.
• That tiny runtime check is overlapped with the GPU's forward pass, so the constraint adds almost nothing to wall-clock latency.

The counterintuitive result, echoed by local runtimes like Ollama's format parameter (see Ollama structured outputs), is that constrained generation is often faster than free generation — the model never wastes tokens deliberating about formatting or wrapping JSON in prose.

What it looks like in practice

vLLM / SGLang — pass the schema as a guided-decoding option:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="x")

resp = client.chat.completions.create(
    model="local-model",
    messages=[{"role": "user", "content": "Extract the invoice."}],
    extra_body={
        "guided_json": {
            "type": "object",
            "properties": {
                "invoice_id": {"type": "string"},
                "total": {"type": "number"},
                "paid": {"type": "boolean"}
            },
            "required": ["invoice_id", "total", "paid"],
            "additionalProperties": False
        }
    },
)

llama.cpp with GBNF — the grammar can be written directly, or auto-converted from a JSON Schema:

root   ::= "{" ws "\"total\"" ws ":" ws number ws ","
           ws "\"paid\"" ws ":" ws ("true" | "false") ws "}"
number ::= "-"? [0-9]+ ("." [0-9]+)?
ws     ::= [ \t\n]*

OpenAI / Anthropic — the same idea, hosted: OpenAI's Structured Outputs with a strict schema enforces the shape by constrained decoding and reports sub-0.1% failure rates; Anthropic's tool-use path is equivalent. You supply a schema, the provider guarantees the parse. See function calling & JSON Schema.

Where constrained decoding still bites

Guaranteed-parseable is not guaranteed-correct, and the constraint is not free of side effects.

• The "constraint tax." Forcing the token distribution through a narrow grammar can *lower answer quality* on some open-weight models — most notably it can suppress a model's tendency to call tools or reason before answering. If accuracy drops when you turn on strict mode, this is why; give the model a free-text reasoning field *before* the structured fields so it can think inside the schema.
• Valid shape, wrong values. The grammar enforces types and structure, not semantics. A total of 0 is schema-valid and business-wrong. You still need to validate against a stricter schema and check ranges, enums, and cross-field rules.
• Not every constraint is expressible. Regex-based engines can't express arbitrary recursion; some engines don't support every JSON Schema keyword. Unsupported keywords are silently ignored, so read your engine's compatibility notes.
• Tokenizer alignment. The grammar operates on the model's exact tokenizer. Point a grammar compiled for one tokenizer at a different model and the masks are wrong.

Inspecting constrained output

When a constrained response is valid JSON but the *content* is off, treat it like any other payload:

1. Format the output so nested extraction results are readable.
2. Diff a good and a bad run to isolate which field the model got wrong under the constraint.
3. Generate a tighter JSON Schema from a known-good sample, then feed that back as the constraint — a narrower grammar leaves the model less room to be wrong.
4. For large results, explore the tree instead of scrolling raw text.

For the neighboring topics, see Validate LLM JSON against a schema, Fix broken JSON from LLMs (the retry-based world constrained decoding replaces), and Streaming partial JSON.