A model must return output conforming to a JSON Schema, but occasionally emits syntactically valid JSON with an invalid enum or missing field. When would you use constrained decoding, schema validation with retries, or both, and what are the latency and availability trade-offs?

How to think about it

The short answer

Use constrained decoding when the output shape is a hard contract and your model-serving stack can enforce the relevant parts of the schema. In production, I would usually combine it with post-generation schema validation, then allow one bounded retry for recoverable failures rather than trusting either mechanism alone.

Why the distinction matters

Imagine an inbox-triage service. It must decide whether a message is about billing, a technical problem, or an account; assign a priority; and say whether a human reply is needed.

The model returns this:

{"priority":"urgent","reply_needed":true}

That is syntactically valid JSON. A JSON parser is happy because the quotes, colon, comma, and braces are in the right places.

It is still an invalid application response. The category field is missing, and urgent is not one of the allowed priority values.

A JSON Schema is a declarative contract describing those rules:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "additionalProperties": false,
  "properties": {
    "category": {
      "type": "string",
      "enum": ["billing", "technical", "account"]
    },
    "priority": {
      "type": "string",
      "enum": ["low", "normal", "high"]
    },
    "reply_needed": {
      "type": "boolean"
    }
  },
  "required": ["category", "priority", "reply_needed"]
}

The interviewer is probing whether you understand that parsing and validation happen at different layers.

Parsing asks, “Can I read this as JSON?” Validation asks, “Does this parsed value satisfy the contract?” The first is necessary. It is not sufficient.

What constrained decoding actually does

Constrained decoding changes generation itself. The decoder tracks which tokens are legal at each point and masks out illegal next tokens.

After the model has generated the key priority, for example, the decoder can permit only values that lead to low, normal, or high. It can also prevent the model from ending the object before all required fields have appeared, assuming the implementation supports those schema rules.

That attacks the problem before the bad value exists. It is generally more reliable than asking the model to “please return valid JSON” and hoping its manners have improved.

But “constrained decoding” is not one universal guarantee. Serving systems support different subsets of JSON Schema. One may enforce object shape and enums but not every keyword, recursive reference, numeric boundary, or conditional rule. Some products call a feature “structured output” while using a mixture of grammar constraints and model training. The exact guarantee belongs to the current provider documentation and your own tests.

A constrained decoder can also enforce only what is expressible in the generation grammar. It cannot know whether customer_id exists in your database, whether a date is a public holiday, or whether the requested refund is legally permitted. Those remain post-generation checks.

What validation with retries does

The alternative is simpler:

1. Generate ordinary output.
2. Parse it.
3. Validate the parsed value against the schema.
4. If it fails, retry with a bounded budget.
5. If it still fails, return an explicit error, quarantine the item, or use a safe fallback.

Validation is usually cheap compared with a model call. It also gives you one consistent trust boundary regardless of how the response was generated.

A retry can include the validation error, such as “category is required and priority must be one of three values.” That often fixes a transient omission. It is not magic, though. If the retry repeats the same prompt, model, temperature, and seed, the second response may repeat the same mistake. Temperature zero means repeatable, not correct.

Do not silently turn urgent into high unless that mapping is an explicit business rule. Otherwise the system has converted a visible model failure into an invisible data error, which is how bad dashboards acquire a calm expression.

Choosing among the three patterns

Pattern	Latency	Availability and failure behavior	Best fit
Constrained decoding	One model call, plus grammar or schema setup overhead	Fewer malformed responses, but unsupported or over-complex schemas can make the request fail before generation	Hard machine-readable contracts
Validation plus retry	Fast on the successful path; failures add a full model round trip	Extra calls consume capacity and can amplify overload; bounded retries make behavior predictable	Providers without usable constraints, or low invalid-output rates
Both	Constraint overhead plus local validation; usually avoids retries	Strongest contract boundary, with a fallback path for provider constraint failures	Payments, workflows, tool calls, and other high-consequence paths

The important latency distinction is between average latency and tail latency. Consider a service with a measured 700 millisecond model round trip and 5 milliseconds of local validation. If 4 percent of ordinary responses fail validation and you allow one retry, the average cost is roughly one extra 700 millisecond call for 4 percent of requests: about 733 milliseconds before other network time. More importantly, the failed requests occupy the slow tail, so p95 can move toward two model round trips.

If those failures are independent, one retry could reduce a 4 percent failure rate toward 0.16 percent. If the failures come from a deterministic prompt defect or an unsupported schema, the retry rate may be nearly unchanged. You have paid twice and learned nothing.

Constrained decoding often gives better tail behavior because it prevents common invalid outputs without a second model call. It can have a cold-start cost when the server compiles the schema or grammar, and a cached schema may be cheaper than a new schema on every request. Measure that setup cost in your workload rather than quoting a universal number; schema size, provider, model, and cache behavior all matter.

Availability means more than “the model endpoint answered.” It means the system produced a usable result within its deadline. Retries improve recovery from occasional bad generations, but they also increase traffic. At 100 requests per second, a 4 percent retry rate adds about 4 model requests per second. During an outage or rate-limit event, that extra load can make the outage worse.

Constrained decoding has its own availability risk. A provider may reject a schema it cannot compile, return an error after a platform update, or expose a narrower feature set than expected. A sensible client treats constraint setup failure as a first-class error. It does not quietly assume that ordinary text generation still satisfies the contract.

The production answer

For a hard contract, my default is:

• Use constrained decoding for the generation call.
• Parse and validate the returned bytes anyway.
• Retry at most once, within a clear deadline, if the response is invalid and a retry is likely to help.
• Record whether failure happened during schema setup, parsing, validation, timeout, or rate limiting.
• Never pass an invalid object to the downstream tool or write path.

The validator remains necessary because it protects against implementation gaps, provider regressions, accidental changes to the schema, and code paths that bypass constrained generation.

For a lower-stakes task, such as extracting rough labels for an analyst queue, ordinary generation plus validation and one retry may be the better trade. It avoids provider-specific constraint limitations and may be cheaper when invalid outputs are genuinely rare.

For a payment instruction or an automated account change, I would not use an unconstrained fallback merely to preserve availability. A rejected request or human review is preferable to a syntactically neat, semantically wrong command.

A failure mode you should expect

The first symptom of an unhealthy retry design is often not invalid JSON. It is a latency and capacity problem: normal p50 latency stays near 700 milliseconds, p99 climbs above 2 seconds, and the service starts returning rate-limit errors.

That pattern usually means validation failures are triggering model calls faster than the system can absorb them. Look at retry counts by schema version and model version. If the same enum failure appears after every retry, stop retrying that class of error and fix the prompt, schema support, or routing logic.

What they’ll ask next

Can constrained decoding guarantee that every JSON Schema is satisfied?
No. It guarantees only what that implementation supports and enforces. Validate the final object, and test the exact schema features you depend on.

How many retries would you allow?
Usually zero or one for a user-facing request, with a hard deadline. More retries can improve recovery in theory but create queueing, rate-limit, and tail-latency problems. A retry should change something meaningful, such as including the validation error or using a known-good fallback model.

What if the provider does not support constrained decoding?
Use ordinary generation followed by strict parsing and schema validation. Retry once if the failure is plausibly transient. For a hard contract, quarantine or fail closed when the retry fails; do not send an unvalidated object downstream.

One line to use in the room

“I use constrained decoding to prevent invalid continuations, but I still validate at the boundary; retries are a bounded recovery mechanism, not a substitute for either enforcement or validation.”