How do you reliably get structured outputs (JSON, typed objects) from an LLM?
Modern APIs offer constrained decoding — the model's token sampling is restricted to only produce tokens that are valid continuations of a JSON schema. Combined with Pydantic validation in application code, this eliminates the JSON-parsing errors that plagued earlier prompt-only approaches. When constrained decoding is unavailable, few-shot examples plus output parsing with retry is the fallback.
How to think about it
Structured outputs matter any time the LLM result feeds into downstream code — parsers, databases, APIs. Prose answers are fine for humans; structured outputs are required for machines.
Approach 1 — Constrained decoding via API (preferred)
OpenAI’s Structured Outputs mode and Anthropic’s tool-use schema enforce grammar constraints at the logit level. The model is physically unable to emit a token that breaks the schema.
from openai import OpenAI
from pydantic import BaseModel
class ProductReview(BaseModel):
sentiment: str # "positive" | "neutral" | "negative"
score: int # 1–5
key_points: list[str]
client = OpenAI()
completion = client.beta.chat.completions.parse(
model="gpt-4o",
messages=[
{"role": "system", "content": "Extract review data."},
{"role": "user", "content": "This laptop is blazing fast but the battery is average. 4/5."},
],
response_format=ProductReview,
)
review: ProductReview = completion.choices[0].message.parsed
print(review.sentiment, review.score)Approach 2 — Tool-call schema (Anthropic)
Define the desired structure as a tool schema. The model always calls the tool, guaranteeing structured output even without constrained decoding.
import anthropic, json
client = anthropic.Anthropic()
response = client.messages.create(
model="claude-opus-4-5",
max_tokens=512,
tools=[{
"name": "submit_review",
"description": "Submit extracted review data.",
"input_schema": {
"type": "object",
"properties": {
"sentiment": {"type": "string"},
"score": {"type": "integer"},
},
"required": ["sentiment", "score"],
},
}],
tool_choice={"type": "tool", "name": "submit_review"},
messages=[{"role": "user", "content": "Great product, minor flaws. 4 stars."}],
)
data = response.content[0].inputApproach 3 — Prompt + parse + retry (fallback)
Instruct the model to return valid JSON, attempt json.loads, and retry with an error message injected on failure. Reliable for simple schemas but fragile for deeply nested structures.
Validation layer
Always validate with Pydantic regardless of the API approach — constrained decoding can still produce semantically invalid values (e.g., score: 99).