datarekha
NLP & LLMs Medium

What is constrained decoding and how does it guarantee structured outputs like valid JSON?

For AI / LLM EngineerML Engineer
The short answer

Constrained decoding masks the model's next-token logits at each step so only tokens permitted by a grammar or JSON schema can be sampled, guaranteeing structurally valid output without changing the model's weights. It is how structured-output and function-calling features enforce schema conformance; placing reasoning fields before answer fields lets the model think before it commits.

How to think about it

Constrained decoding masks the model’s next-token logits at each step so only tokens permitted by a grammar or JSON schema can be sampled, guaranteeing structurally valid output without changing the model’s weights. It is how structured-output and function-calling features enforce schema conformance; placing reasoning fields before answer fields lets the model think before it commits.

Learn it properly Constrained decoding

Keep practising

How do you reliably get structured outputs (JSON, typed objects) from an LLM? What is the difference between encoder-only, decoder-only, and encoder-decoder transformer architectures? What is the difference between encoder-only, decoder-only, and encoder-decoder transformer architectures? What is the difference between encoder models like BERT and decoder models like GPT? What is a data contract, and how does it prevent ML pipelines from breaking silently?
All NLP & LLMs questions

Explore further

Structured outputs Speculative Decoding Pydantic for LLM outputs
Constrained Decoding Decoder Encoder GPT
Skip to content