Constrained decoding

At 3:07 a.m., your support-ticket pipeline receives this:

{"category":"billing","priority":5,"summary":"Customer says their card was charged twice",}

The answer is understandable. It is also useless to the next service. The trailing comma makes the JSON parser reject the entire record. A retry might fix it. A second retry might produce Python’s True instead of JSON’s true. Meanwhile, the ticket waits.

Asking a language model to “respond in JSON” usually works. Usually is not a reliability property.

The structured outputs lesson covered the public promise: output that follows a schema. This lesson explains the mechanism underneath that promise. Constrained decoding makes some continuations unavailable to the model before it chooses its next token. It does not persuade the model to behave. It removes illegal choices.

Prompting is a hope; constraining is a guarantee

A language model generates one token at a time: a small piece of text such as {", billing, or True. At each step it produces a logit, an unnormalised score for every vocabulary token. Sampling turns those scores into a choice.

Normally, every vocabulary token is eligible. The model can close an object too early, put a comma where a value belongs, or emit True, which is valid Python but not JSON.

Constrained decoding inserts a grammar between the model and the sampler:

1. The model produces logits for the next token.
2. The grammar examines the emitted prefix.
3. It returns tokens legal in the current grammar state.
4. The decoder sets every other token’s logit to -infinity.
5. Sampling chooses from the survivors.
6. The grammar advances according to the chosen token.

A probability of zero is stronger than a low probability. Temperature, top-k, and top-p sampling cannot select a token that was masked before sampling.

A grammar is often introduced as a finite-state machine, but nested objects need stack-like tracking, and schemas may track which optional or required fields have appeared. Serving engines compile the schema into a representation that can handle those transitions.

The simple JSON path looks like this:

At a general value position, the grammar can allow a JSON string, number, boolean, null, object, or array. At a schema-specific position it can be narrower: an enum containing "billing" and "technical" excludes numbers, while an integer constrained to 1–5 excludes 9.

That is the boundary of the guarantee. Constrained decoding enforces the language of the output, not the truth of its claims.

A running example: extracting a support ticket

Suppose the model reads an incoming email and must produce:

{
  "category": "billing",
  "priority": 5,
  "summary": "Customer says their card was charged twice"
}

The application requires category to be "billing" or "technical", priority to be an integer from 1 through 5, and summary to be a nonempty string. All three fields must appear, with no fourth field.

A JSON Schema describes those requirements:

{
  "type": "object",
  "properties": {
    "category": {
      "type": "string",
      "enum": ["billing", "technical"]
    },
    "priority": {
      "type": "integer",
      "minimum": 1,
      "maximum": 5
    },
    "summary": {
      "type": "string",
      "minLength": 1
    }
  },
  "required": ["category", "priority", "summary"],
  "additionalProperties": false
}

The compiler turns the schema into grammar states. At the start, only an opening object token is legal. After {, it permits an allowed property name. After "category", it permits :, then the beginning of one of the two enum strings. An arbitrary key is never legal because additionalProperties is false. A closing brace is legal only after all required fields have appeared.

The model may strongly prefer "shipping". The grammar does not correct that preference; it gives the continuation a zero score and leaves only legal alternatives.

Tokenization makes this more involved than checking one character. A token can contain several characters or span several grammar transitions. For priority, a token for 12 must be rejected as a whole, while a token for 1 may be accepted because it is a complete allowed value. A real decoder checks whether the entire candidate token can be consumed and advances to the resulting grammar state.

The mask, with actual numbers

Here is a deliberately small vocabulary at a generic JSON value position:

import numpy as np

vocab = ["true", "True", '"yes"', "1", ",", "}"]
logits = np.array([2.1, 2.4, 1.0, 0.3, 0.5, 0.2])         # the model "prefers" True

# The grammar says: at a value position, only these tokens are legal.
legal = {"true", '"yes"', "1"}
mask = np.array([t in legal for t in vocab])

def softmax(x):
    e = np.exp(x - x.max()); return e / e.sum()

print("unconstrained pick:", vocab[logits.argmax()], "(invalid JSON!)")

masked = np.where(mask, logits, -np.inf)                  # illegal logits -> -inf
probs = softmax(masked)
print("constrained pick:  ", vocab[int(probs.argmax())], "(valid)")
print("masked probabilities:", {v: round(float(p), 3) for v, p in zip(vocab, probs)})

unconstrained pick: True (invalid JSON!)
constrained pick:   true (valid)
masked probabilities: {'true': 0.667, 'True': 0.0, '"yes"': 0.222, '1': 0.11, ',': 0.0, '}': 0.0}

Before masking, True wins with logit 2.4, just above true at 2.1. After masking, softmax renormalises the three legal logits: true gets about 0.667, "yes" 0.222, and 1 0.110. The invalid token is not lowercased or repaired; its probability is zero before selection.

Production tokenizers may contain hundreds of thousands of multi-character tokens, so engines often cache which tokens are valid from each grammar state. Compilation and vocabulary traversal, rather than the small mask itself, are common sources of overhead.

What the guarantee really means

For a supported grammar, every emitted token keeps the prefix within the grammar’s viable-prefix set. If decoding ends in an accepting state, the completed text belongs to the grammar’s language. It does not mean that the answer is useful, true, safe, or complete.

For the ticket schema, constrained decoding can guarantee:

• the output parses as JSON;
• required keys exist;
• category is an allowed enum value;
• priority is within its declared range;
• no undeclared key appears, if that restriction is supported and enabled.

It cannot guarantee that the email describes billing, that priority 5 is justified, or that the summary preserves every important detail. It also cannot authorise a tool call.

A model can return this perfectly valid but wrong record:

{
  "category": "billing",
  "priority": 5,
  "summary": "Customer says the card was charged once"
}

A parser will celebrate.

Completion is another limit. If the output budget ends inside the summary, every emitted prefix may still be grammar-valid, but the response is not a complete JSON document. The runtime must report truncation or failure; callers should inspect completion and refusal status before accepting a record.

Schema support also varies. An engine may support enums and required fields but not every conditional, recursive, or numeric JSON Schema feature. Some reject unsupported keywords; others simplify them. Check the provider or server documentation and test the constraints your application depends on.

JSON mode, structured output, and a grammar

These names are related but not interchangeable:

• Prompt-only JSON asks the model to follow a format. It has no structural guarantee.
• JSON mode generally targets parseable JSON, but may not enforce field names, types, required fields, or enums.
• Structured output usually means schema-aware constraints.
• A custom grammar can constrain another formal language, such as a command language or domain-specific record.

Even schema-constrained output needs ordinary application validation. Use code for ranges, cross-field rules, permissions, and external state. A ticket can satisfy its schema while violating a rule such as “priority 5 requires evidence of an outage.”

The production pattern

A dependable implementation has five layers:

1. Make the schema narrow. Use enums instead of unconstrained strings, set numeric bounds, require fields downstream services need, and disallow additional properties when appropriate. Include "other" or a refusal state when the enum would otherwise force an inaccurate classification.
2. Compile and cache the grammar. Schema compilation and tokenizer-specific mask preparation can make the first request much slower. Reuse compiled grammars by schema and tokenizer where possible.
3. Use the real structured-output interface. Confirm whether the API guarantees JSON syntax, the full schema, or only best effort. Check how it reports refusal, truncation, unsupported features, and a state with no legal tokens.
4. Parse and validate at the application boundary. This catches misconfiguration, schema-version mismatches, manually produced data, and streams cut off in transit.
5. Apply semantic and security checks. For a refund tool call, verify identity, ownership, amount, current state, and permission in ordinary code. Record schema version, completion status, validation results, and enum distributions for evaluation.

When it helps, and when it hurts

Use constrained decoding when another program consumes the output: tool arguments, extraction records, API payloads, routing decisions, database writes, and agent actions. Do not constrain ordinary prose unless a machine truly depends on its shape.

Constraints can force the model into the nearest legal answer when the desired answer is not representable. If the category enum contains only "billing" and "technical", an account-deletion email must be assigned one of them unless the schema offers "other" or refusal. A badly designed schema can therefore make output more consistently wrong.

There is also a latency and quality cost. Mask computation is often modest, but large grammars, long enums, cold compilation, or short generations can make it visible. Measure time to first token and total generation time on your workload. The final answer is the highest-scoring legal continuation, not necessarily what the model would have written freely.

Failure modes you can diagnose

The key diagnostic split is syntax failure versus semantic failure. Adding grammar to the second problem usually produces a more elegantly wrong answer.

Quick check

Next

Constrained decoding is the reliability layer beneath tool calling and structured agent actions. To check whether the content inside that valid structure is actually right, pair it with LLM evals.