AI Structured Outputs Explained: JSON Mode, Schema Enforcement, Reliable Parsing

Structured outputs are how you stop guessing whether the model will return valid JSON. Instead of writing a paragraph asking for “JSON only” and hoping for the best, you give the model a JSON Schema that constrains every response, and the API rejects completions that would not parse. The result is the same reliability you get from a typed function call — your downstream code can trust the shape, your pipelines do not break on stray prose, and your error handling gets simpler.

This guide explains what structured outputs and JSON mode actually do, how the three major providers implement them differently, how to design schemas that are both expressive enough to be useful and strict enough to be enforced, and the failure modes (refusals, max_tokens truncation, refusals-as-content) that catch teams when they first switch from “free-form prompts asking for JSON” to enforced schemas. The pattern pairs naturally with function calling and benefits from the same OpenAI-compatible contract if you want to swap providers without rewriting your code.

Why structured outputs matter

Most production AI features pass model output into something else: a database, a UI, a downstream function, an analytics pipeline. The moment the output becomes data rather than text, its shape matters — a missing field or a typo in a key can break every consumer downstream. “Just ask for JSON” is the standard solution, but it is fragile: the model can still return "Sure, here is the JSON: {...}", wrap the object in an array, escape characters incorrectly, or halluc additional fields. Code that consumes that output has to be defensive in ways that turn simple tasks into messy parsers.

Structured outputs fix this at the source. You supply a JSON Schema; the provider’s API guarantees the response matches it. Your code becomes a typed function call: resp.parsed is a Python object, not a string to wrangle. Reliability goes up, complexity goes down, and the difference is felt most at the edges — when the prompt is ambiguous, when the model is small, when you switch versions and behaviour shifts.

JSON mode vs structured outputs

These two terms sound similar but they guarantee different things:

• JSON mode ensures the output is valid JSON. The model still chooses the shape — you cannot pin down which keys appear or what their types are. Useful when you want a JSON object but the schema is trivial or up to the model.
• Structured outputs enforce a specific JSON Schema. The provider rejects any completion that does not match the schema’s structure, required fields, and enum constraints. Use this whenever downstream code assumes a particular shape.

JSON mode is the older, weaker guarantee. Structured outputs are what you want in production. Most modern providers now ship some form of structured outputs, but the level of strictness varies — and that variation is where most integration bugs live.

Defining a usable schema

The schema you write is both a contract and a constraint. A good one describes the data you need, nothing more — and avoids features the provider cannot enforce. Three rules consistently produce schemas that work:

1. Require every field. Marking a property as required is the only way to guarantee it appears. Optional fields should have an explicit null default in the type list (e.g. "type": ["string", "null"]) and be clearly marked.
2. Use enums for closed sets. When a field can only take a few values, declare them with enum. This is the single biggest quality improvement available, because the model no longer has to invent plausible-sounding strings.
3. Keep descriptions short and descriptive. The description is what the model reads to decide what value to produce. “The customer’s sentiment in one word” is more useful than a paragraph.

A complete OpenAI structured-outputs example

OpenAI’s structured outputs use a response_format with type: "json_schema" and a JSON Schema that is constrained to a subset the API can enforce. The strict: true flag is what turns the schema into a hard guarantee:

from openai import OpenAI
from pydantic import BaseModel

client = OpenAI()

class Sentiment(BaseModel):
    label: str          # "positive" | "neutral" | "negative"
    score: float        # 0.0 .. 1.0
    summary: str        # one sentence

SCHEMA = {
    "type": "json_schema",
    "json_schema": {
        "name": "sentiment",
        "strict": True,
        "schema": Sentiment.model_json_schema(),
    },
}

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": "Classify the sentiment of the review."},
        {"role": "user",   "content": "I waited three weeks and it never arrived."},
    ],
    response_format=SCHEMA,
)

result = Sentiment.model_validate_json(resp.choices[0].message.content)
print(result.label, result.score, result.summary)

Two details matter. First, strict: true rejects any completion that would not parse — you either get a fully-formed object or an error, never a half-formed one. Second, the first call with a new schema incurs a small one-time cost as the provider compiles the grammar; subsequent calls are fast.

Cross-provider differences

If you call more than one provider, normalise the schema shape and let a small adapter translate it. The patterns look similar; the field names do not:

Concept	OpenAI	Anthropic Claude	Google Gemini
JSON mode	response_format: {"type": "json_object"}	No native mode; prompt + JSON syntax in output	generation_config.response_mime_type = "application/json"
Structured outputs	response_format.type = "json_schema" with strict: true	Tools (function calling) acts as a strict schema	generation_config.response_schema
Schema language	JSON Schema (subset)	JSON Schema (tool input)	OpenAPI 3 subset
Refusal handling	message.refusal field	Stop reason; no special field	Finish reason SAFETY

The most useful abstraction in a multi-provider stack is a “schema-bearing request”” that compiles to each provider’s specific shape. Our guide to the OpenAI-compatible API explains why this is exactly the kind of variation a gateway is designed to absorb.

Structured outputs vs function calling

Both technologies constrain a model’s output, but they solve different problems. Choose based on intent:

• Use structured outputs when the model should answer with a structured value — classification, extraction, scoring, summarisation into a record. The model’s output is the data.
• Use function calling when the model should act by requesting that your code run something — fetching a record, calling another API, querying a database. The model’s output is a request; your code does the real work.

It is common to combine both: a function-calling tool whose arguments are themselves validated against a JSON Schema, plus the final assistant response wrapped in structured outputs for safe parsing. The two compose cleanly because both share the same underlying contract.

Failure modes that catch teams

Even with strict schemas, three failure modes recur in production:

• Refusals. When the model refuses to answer (safety filter, content policy, ambiguous prompt), the provider returns a refusal object rather than a completion that matches the schema. Code that only checks for a parsed object will mis-handle it. Always inspect the refusal field before validating the body.
• max_tokens truncation. If a completion runs out of tokens mid-object, you get a syntactically broken string. Set max_tokens generously, validate before storing, and treat “unparseable output” as a retry signal — sometimes a larger budget alone solves it.
• Unsupported schema features. Each provider enforces a subset of JSON Schema. OpenAPI-style formats ("format": "date-time"), recursive references, and arbitrary unions are commonly restricted. When the API rejects a schema, the error is usually specific enough to point at the feature — read it carefully and simplify rather than fighting the provider.

Best practices for production

Three habits keep structured outputs reliable in the long run:

• Define schemas in code, not prompts. Use Pydantic, Zod, or similar to derive the JSON Schema from a typed class. The model of “write JSON in the prompt and pray” is what structured outputs replaces — keep the prompt focused on intent and let the schema enforce the shape.
• Validate at the boundary, trust inside. Once the response has parsed, treat it as typed data. Do not defensively re-validate every field in business logic; that adds noise and loses the value of the type.
• Version your schemas. When the contract changes, old prompts and old responses may not match the new shape. Either ship the new schema as a separate endpoint, or version the response wrapper and migrate callers together.

Structured-outputs checklist

• Derive the schema from a typed class (Pydantic, Zod) rather than hand-writing JSON.
• Mark every field as required unless you genuinely want it optional.
• Use enum for closed sets and short descriptions for everything else.
• Always check the refusal field before validating the body.
• Set max_tokens generously enough for the longest expected output.
• Treat unparseable output as a retry signal — same input, larger budget if needed.
• Stay within the provider’s supported schema subset; simplify on rejection.
• Version the schema and migrate callers together when the shape changes.
• If you call multiple providers, normalise via an adapter, not by hand in each call site.

Frequently asked questions

What is the difference between JSON mode and structured outputs?

JSON mode guarantees the response is valid JSON. Structured outputs add a JSON Schema guarantee: the keys, types, enums, and required fields are enforced. In production, structured outputs are almost always what you want — JSON mode alone is a weak guarantee.

Does the model always respect the schema?

With strict: true (or the equivalent flag in each provider), the API rejects completions that would not match the schema before they reach your code. You either get a fully-formed object or an error — never a half-formed one. This is the practical difference from prompting for JSON in plain text.

Which providers support structured outputs?

OpenAI has the strictest implementation under response_format.type = "json_schema" with strict: true. Google Gemini supports an OpenAPI subset via response_schema. Anthropic Claude does not expose a native JSON-mode flag, but tool-calling arguments are validated against a JSON Schema and act as a strict contract. If you call more than one, write a small adapter that compiles your schema into each provider’s format — see our OpenAI-compatible API guide for the wrapper tradeoffs.

What happens when the model refuses to answer?

It does not produce a partial object — it produces a refusal object alongside (or instead of) the structured content. Code that calls model_validate_json on a refusal body will fail. Always check the refusal field first, then parse. Treating refusals as a separate outcome rather than an exception keeps your error handling clean.

What if my schema is too complex for the provider?

Each provider enforces a subset of JSON Schema. Recursive references, arbitrary unions, and certain format keywords are commonly restricted. When the API rejects a schema, the error usually names the unsupported feature. Simplify the schema, or split a complex value into two calls — a strict schema for the parts you can enforce and a free-form string field for the parts you cannot.

Do structured outputs cost more than regular completions?

Most providers do not charge more for the schema itself, but you still pay for the input and output tokens it produces. The first call with a new schema can incur a small one-time cost as the provider compiles a grammar. Subsequent calls are essentially the same cost as a regular completion. See our guide to reducing AI API costs for related cost patterns.

Should I use structured outputs or function calling?

Use structured outputs when the model should answer with data — classification, extraction, scoring, summarisation. Use function calling when the model should act by requesting that your code do something. The two compose: a tool’s arguments are themselves validated against a JSON Schema, and the final assistant response can also be structured. See our guide to function calling for the action side of this pattern.

---

Structured outputs are the production-grade answer to “give me JSON”. With a typed schema, the API guarantees the shape; with a refusal field, your error handling stays clean; with versioned schemas, your contracts evolve without surprises. The investment is small — most teams retrofit an existing prompt in under an hour — and the payoff is felt everywhere the model’s output becomes data. If you want to try it against multiple models without rewriting your client, create a key at qoraapi.com and your code can speak OpenAI’s response_format against GPT, Claude, or Gemini.

Related reading

• AI Function Calling Explained: Tools, JSON Schema, and the Tool-Use Loop
• Extracting Structured Data from Documents with AI APIs
• Text-to-SQL: Letting Users Query Your Database with AI
• Detecting and Reducing Hallucinations in Production LLM Apps
• Building Voice AI Apps: TTS, STT, and Realtime APIs
• How to Choose a Vector Database for RAG
• Managing the Context Window: Truncation, Summarization, and Sliding Windows