What is constrained decoding and how does it guarantee structured outputs like valid JSON?

How to think about it

Constrained decoding is a generation-time guardrail that removes illegal next-token choices according to a grammar or JSON Schema. It can guarantee that a completed response is valid JSON, and can also guarantee supported schema rules, but it cannot guarantee that the model’s values are truthful or useful.

The mechanism an interviewer wants

An LLM is autoregressive, meaning it generates one token at a time, using everything already generated as context. A token is a piece of text chosen by the model’s tokenizer; it might be a whole word, part of a word, punctuation, or several characters. At each step, the model produces logits, which are raw scores for possible next tokens.

Normally, the decoder turns those scores into probabilities and chooses the next token by sampling or by taking the highest-scoring option. Nothing stops the model from producing a missing quote, an extra comma, or the word urgent when the application only accepts low, normal, or high.

A constrained decoder maintains the current state of a parser. A parser is a piece of software that knows which characters or tokens can legally come next. The decoder asks it for the allowed continuations, masks every other token by assigning it effectively -infinity, and then samples only from what remains. The model weights do not change. The set of available choices does.

The constraint is applied at every step, not after generation. That distinction matters. A post-processing validator can notice that the model produced broken JSON, but it cannot prevent the broken output. A repair step may fix the comma while accidentally changing a value. Constrained decoding prevents the illegal token from being emitted in the first place.

Tokenization is the detail that catches many otherwise good answers. Constraints cannot safely operate only one character at a time because the model chooses tokens. At the prefix {"priority":"h, the next token must leave a prefix that can still become a valid value such as "high". A token that would complete "urgent" is rejected, even if its first character happens to look plausible. Production decoders therefore need to be tokenizer-aware.

A JSON grammar can enforce syntax: balanced braces, quoted keys, escaped strings, commas in the right places, and legal values for booleans and numbers. A schema-aware decoder adds rules such as required properties, enumerated values, types, and sometimes numeric bounds. The model still chooses among legal options, so the output can vary. It simply cannot vary outside the contract.

A concrete example

Imagine classifying this support ticket:

Ticket A-1042 says, “I was charged twice for my annual subscription. Please reverse one of the charges.”

The application wants a small object for its billing workflow:

{
  "type": "object",
  "properties": {
    "ticket_id": {
      "type": "string"
    },
    "priority": {
      "type": "string",
      "enum": ["low", "normal", "high"]
    },
    "confidence": {
      "type": "number",
      "minimum": 0,
      "maximum": 1
    },
    "needs_human": {
      "type": "boolean"
    }
  },
  "required": ["ticket_id", "priority", "confidence", "needs_human"],
  "additionalProperties": false
}

A valid result could be:

{
  "ticket_id": "A-1042",
  "priority": "high",
  "confidence": 0.97,
  "needs_human": true
}

With the schema rules supported by the decoder, several bad continuations become impossible:

• "priority": "urgent" violates the enumeration.
• "confidence": "0.97" violates the number type because the value is a string.
• "needs_human": maybe is not a JSON boolean.
• An extra "source": "email" property violates additionalProperties: false.
• A closing brace before all required properties have appeared is not an accepting ending.

The decoder does not understand that a duplicate charge is genuinely important. That judgment comes from the model and the prompt or training. It only ensures that the judgment is represented using the permitted shape and values.

That is the central distinction:

Layer	Constrained decoding can enforce	It cannot establish
JSON grammar	Valid quotes, commas, braces, strings, numbers, and booleans	That the content is correct
Schema rules	Supported types, required keys, enums, and bounds	That high is the right priority
Business meaning	Nothing by itself	Whether the customer was actually charged twice

The senior-level nuance

Common misconception: valid JSON is not the same as a valid answer. This object is perfectly valid JSON:

{
  "priority": "high"
}

It may still fail the application’s contract because required fields are missing. Even a schema-conformant object can be wrong. The model could confidently assign high to a harmless password-reset request, or return 0.97 when its classifier is poorly calibrated. Structure is guaranteed; truth is not.

Schema guarantees are also conditional on what the decoding implementation supports. Different products support different JSON Schema dialects and keyword subsets. type, required, enum, and additionalProperties are common. Complex conditionals, custom formats, cross-field relationships, and some regular-expression rules may be rejected, ignored, or enforced only by a later validator. A strong engineer checks the current product documentation and tests the exact schema rather than trusting the label “structured output.”

Completion status matters too. The guarantee applies to a response that reaches a valid ending. If a long explanation consumes the output limit halfway through a string, the stream can end without a closing quote or brace. A streaming client may observe unexpected end of JSON input, even though the decoder never allowed an invalid completed object. Refusals, timeouts, and transport failures also need separate handling; they are not ordinary successful schema results.

Field order deserves a careful answer. LLM output is sequential, so asking for a short rationale before priority can give the model more tokens to analyze before it commits to the final field. But JSON object properties are semantically unordered. Putting rationale first in the required list does not, by itself, force that generation order. An order-sensitive grammar or template is needed. And a rationale field is not a guaranteed window into reliable reasoning. It costs tokens and may expose information that should remain private. For production systems, a short explanation or a separate internal reasoning step is usually safer than blindly requesting chain-of-thought.

There is a performance trade-off. The decoder may need to compile a large schema and check legal token continuations for every generated token. That adds latency and implementation complexity, particularly for deeply nested or recursive structures. A smaller schema is usually faster and easier to evolve. Constrained decoding is most valuable when another program will immediately consume the result, such as tool arguments, extraction records, database updates, or routing decisions. It is unnecessary overhead for an open-ended essay.

A failure mode you should recognize

A common production symptom is that short tickets parse correctly but long tickets end with unexpected end of JSON input. The usual cause is truncation: the model spent its output budget on a verbose rationale and never reached the closing part of the object. Keep the schema compact, cap free-form fields, treat incomplete output as a failed request, and retry or request a smaller result.

The other failure is quieter: parsing and schema validation succeed, but users complain that the system makes bad decisions. That is not a decoding failure. Add semantic checks, evaluate classification quality separately, and monitor calibration if the output contains confidence scores. Constrained decoding protects the interface between the model and the application; it does not replace model evaluation or business validation.

What they’ll ask next

Does constrained decoding prevent hallucinations?
No. It can force the model to return a string in the ticket_id field, but it cannot force that string to refer to a real ticket. It controls shape and supported values, not factual accuracy.

Is this the same as JSON mode?
Not necessarily. Names vary by provider. A JSON mode may guarantee parseable JSON while leaving field names, required properties, or enum values unconstrained. A schema-constrained mode usually adds those rules. The contract must be checked in the current API documentation and with negative tests.

Why validate the result if decoding already guarantees the schema?
Validate at the application boundary anyway. You need to detect unsupported schema features, implementation regressions, incomplete responses, refusals, and semantic violations such as a ticket ID that does not exist. Also record the schema version used for each request so a later schema change does not make old records mysterious.

Say this in the interview: Constrained decoding enforces structure before each token is emitted by masking illegal continuations, so a completed response can be guaranteed valid JSON and supported-schema compliant, but the model can still be wrong inside that valid shape.