LLM Structured Outputs: OpenAI, Anthropic & Gemini

Your JSON is failing in production. Not all the time — just often enough to generate JSONDecodeError noise in your logs, force retry logic that doubles your response times, and occasionally break features when a model update shifts output formatting. If you’re extracting structured data from LLM calls using prompt engineering alone, you’re looking at roughly a 15% failure rate. That’s one in seven requests failing.

LLM structured outputs — native schema enforcement built into the API — drop that failure rate to below 0.1%. All three major providers now support it as of early 2026. But most guides cover one provider, one language, and skip the gotchas that bite you in production.

This post covers all three providers (OpenAI, Anthropic, and Gemini) with working Python and TypeScript examples, explains how constrained decoding actually works, and walks through the schema design decisions that determine whether your output quality goes up or down.

The Real Cost of Getting JSON From LLMs the Wrong Way

The failure modes are worth naming precisely. Your model returns:

• JSON wrapped in markdown code fences: ```json\n{...}\n```
• A field typed as number containing "42" (a string)
• A required field silently omitted
• A trailing comma that makes the output unparseable
• The entire response as prose with JSON buried mid-paragraph

Each failure mode requires different error handling. Most teams end up with layered try/catch logic, a regex stripper for markdown fences, and a retry loop. The retry is the expensive part — when a 500ms LLM call fails and retries, you’ve just paid 1000ms+, and your p95 latency climbs fast.

A 2026 analysis aggregated across production LLM deployments puts JSON prompting at roughly a 15% failure rate, with failures causing JSONDecodeError spam and retries that can double response times. Separately, Hostinger’s 2026 LLM statistics report found that 35% of LLM users cite reliability and inaccurate output as their primary concern with AI-powered applications.

The fix is not better prompting. The fix is constrained decoding.

How Constrained Decoding Works — And Why It Guarantees Schema Compliance

Most guides say constrained decoding “ensures” your schema. Few explain why, which means you can’t reason about edge cases when things go wrong.

Here’s the actual mechanism. When you enable strict structured outputs, the provider compiles your JSON Schema into a finite state machine (FSM). The FSM maps every valid state of your output — which fields have been opened, which types are expected next, where commas and brackets belong.

At each token generation step, the LLM produces probability scores across its entire vocabulary (50,000+ tokens). Normally, it picks the highest-probability token. With constrained decoding, any token that would violate the FSM path gets its probability set to zero before sampling. The model cannot emit an invalid token — not because you asked nicely in the prompt, but because those tokens are mathematically excluded from the distribution.

This is fundamentally different from prompt engineering. A prompt like “respond only in JSON with this schema” influences the model. Constrained decoding enforces it. GPT-4o-2024-08-06 with Structured Outputs scores 100% on complex JSON schema adherence evaluations (OpenAI, 2024).

One trade-off worth knowing upfront: constraining the token distribution can reduce model quality on reasoning-heavy tasks because the model loses freedom to express intermediate thinking in natural language. More on this in the schema design section.

JSON Mode vs. Structured Outputs vs. Tool Use — Know What You’re Using

These three things are not the same, and conflating them is how teams end up thinking they have schema enforcement when they don’t.

JSON Mode guarantees valid JSON syntax. The output will parse with json.loads() — but it might have missing required fields, wrong types, or a completely different structure than expected. Better than nothing, but not production-safe for typed pipelines.

Structured Outputs (with strict mode) actually enforces your schema via constrained decoding. Required fields will be present. Types will match. Enums will only contain declared values. This is what you want.

Tool Use / Function Calling is Anthropic’s primary mechanism prior to their 2026 GA release. It routes model output through a tool definition that includes a JSON Schema. With strict: true on the tool definition, you get FSM-based guarantees — but Anthropic adds 313–346 tokens of system prompt overhead per request when tools are provided. At thousands of daily API calls, that’s a real cost line worth tracking — alongside prompt caching, which can cut your total token spend significantly at scale.

Provider summary as of early 2026:
– OpenAI: .parse() with Pydantic, or response_format with json_schema type and strict: true
– Anthropic: Two modes — JSON Outputs Mode (output_format parameter) and Strict Tool Use Mode (strict: true on tool definitions)
– Gemini: response_schema parameter alongside response_mime_type: "application/json"

Provider Playbook: OpenAI, Anthropic, and Gemini Side-by-Side

OpenAI

OpenAI’s implementation is the most straightforward. The .parse() method accepts a Pydantic model directly and returns a typed response object.

from openai import OpenAI
from pydantic import BaseModel

client = OpenAI()

class ArticleSummary(BaseModel):
    reasoning: str
    title: str
    key_points: list[str]
    sentiment: str

response = client.beta.chat.completions.parse(
    model="gpt-4o-2024-08-06",
    messages=[{"role": "user", "content": "Summarize this article: ..."}],
    response_format=ArticleSummary,
)
result = response.choices[0].message.parsed

In TypeScript with the Vercel AI SDK:

import { openai } from "@ai-sdk/openai";
import { generateObject } from "ai";
import { z } from "zod";

const { object } = await generateObject({
  model: openai("gpt-4o-2024-08-06"),
  schema: z.object({
    reasoning: z.string(),
    title: z.string(),
    key_points: z.array(z.string()),
    sentiment: z.string(),
  }),
  prompt: "Summarize this article: ...",
});

Limits: OpenAI strict mode supports schemas with up to 100 object properties total and up to 5 levels of nesting. additionalProperties must be false.

Anthropic (Claude)

Claude’s structured outputs went GA in early 2026 with two modes. For most use cases, JSON Outputs Mode is simpler and avoids the tool overhead:

import anthropic
import json

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-opus-4-5",
    max_tokens=1024,
    output_format={
        "type": "json",
        "schema": {
            "type": "object",
            "properties": {
                "reasoning": {"type": "string"},
                "title": {"type": "string"},
                "key_points": {"type": "array", "items": {"type": "string"}},
                "sentiment": {"type": "string"}
            },
            "required": ["reasoning", "title", "key_points", "sentiment"]
        }
    },
    messages=[{"role": "user", "content": "Summarize this article: ..."}]
)
result = json.loads(response.content[0].text)

In TypeScript:

import { anthropic } from "@ai-sdk/anthropic";
import { generateObject } from "ai";
import { z } from "zod";

const { object } = await generateObject({
  model: anthropic("claude-opus-4-5"),
  schema: z.object({
    reasoning: z.string(),
    title: z.string(),
    key_points: z.array(z.string()),
    sentiment: z.string(),
  }),
  prompt: "Summarize this article: ...",
});

Key gotchas:
– Token overhead: Tool use mode adds 313–346 tokens per request. Prefer JSON Outputs Mode at scale unless you specifically need tool-based routing.
– Schema compilation latency: The first request with a new schema incurs 100–300ms overhead. The compiled grammar is cached for 24 hours — warm it during deployment, not at the first real user request.

Gemini

Gemini uses the response_schema parameter alongside response_mime_type:

import google.generativeai as genai
from pydantic import BaseModel

genai.configure(api_key="YOUR_KEY")

class ArticleSummary(BaseModel):
    reasoning: str
    title: str
    key_points: list[str]
    sentiment: str

model = genai.GenerativeModel("gemini-2.0-flash")
response = model.generate_content(
    "Summarize this article: ...",
    generation_config=genai.GenerationConfig(
        response_mime_type="application/json",
        response_schema=ArticleSummary,
        property_ordering=["reasoning", "title", "key_points", "sentiment"]
    ),
)

In TypeScript:

import { google } from "@ai-sdk/google";
import { generateObject } from "ai";
import { z } from "zod";

const { object } = await generateObject({
  model: google("gemini-2.0-flash"),
  schema: z.object({
    reasoning: z.string(),
    title: z.string(),
    key_points: z.array(z.string()),
    sentiment: z.string(),
  }),
  prompt: "Summarize this article: ...",
});

Critical gotcha: Gemini has a known property ordering issue. Without explicitly setting property_ordering, field order in the output can differ from your schema definition. This doesn’t break JSON parsing, but it confuses downstream code expecting stable ordering and makes diffs harder to read. Always set it explicitly.

Schema Design That Doesn’t Break Your Model

Getting schema compliance is only half the problem. Schema design directly affects output quality.

Put reasoning fields first. This is the highest-impact change most teams aren’t making. If your schema has a reasoning field and an answer field, put reasoning before answer. With constrained decoding, the model generates fields in order — if it commits to an answer first, it hasn’t reasoned yet. The reasoning field becomes post-hoc justification. Flip the order and you get actual chain-of-thought before the answer lands.

Keep nesting to 3 levels or fewer. Deeply nested schemas don’t just hit provider limits — they degrade quality. The model loses track of its position in the schema at depth 4+. Flatten where you can.

Write field descriptions as embedded prompts. The JSON Schema description field is read by the model. It influences output quality. Compare:
– "description": "The sentiment" — useless
– "description": "The overall emotional tone. One of: positive, negative, neutral, mixed. Judge by the author's conclusion, not the subject matter itself." — this is a prompt fragment embedded in your schema

Handle nulls explicitly. Mark optional fields with "type": ["string", "null"]. Implicit nullability confuses models and causes type errors downstream.

Gemini and reasoning quality: A benchmark shows Gemini’s constrained decoding scores 86.18% on the Shuffled Objects reasoning task versus 97.15% for natural language output (dylancastillo.co). For reasoning-heavy schemas on Gemini, consider a two-step approach: get natural language output in the first call, then parse it into your schema in a second extraction call.

When to Use a Library Instead of the Native API

Native APIs are sufficient for single-provider, simple-schema use cases with no additional validation logic.

Use Instructor (Python) when:
– You need the same code to work across multiple providers
– You need cross-field validation (e.g., end date must be after start date)
– You want automatic retry-with-feedback when validation fails — Instructor passes the validation error back to the model and asks it to self-correct

Instructor has 11,000+ GitHub stars and 3 million+ monthly downloads, supporting 15+ LLM providers. The API surface is identical across providers:

import instructor
from anthropic import Anthropic
from pydantic import BaseModel, field_validator

client = instructor.from_anthropic(Anthropic())

class ArticleSummary(BaseModel):
    reasoning: str
    title: str
    key_points: list[str]
    sentiment: str

    @field_validator("key_points")
    def at_least_one_point(cls, v):
        if len(v) < 1:
            raise ValueError("Must include at least one key point")
        return v

result = client.messages.create(
    model="claude-opus-4-5",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Summarize this: ..."}],
    response_model=ArticleSummary,
)

Use Vercel AI SDK + Zod (TypeScript) for the same reasons on the TypeScript side. The generateObject function normalizes the API across OpenAI, Anthropic, and Gemini — switching providers is a one-line change in the model field.

The native API makes sense when you’re on a single provider, have simple schemas, and want zero added dependencies. Otherwise, the library wins on portability and retry logic alone.

Production Hardening Checklist

Native structured outputs shift what you validate, not whether you validate.

• Warm schema caches at deployment. Anthropic’s 100–300ms compilation overhead hits on the first request per schema. Fire a dummy warm-up call during your deployment health check — not when the first real user hits the endpoint.
• Always validate with Pydantic/Zod even with strict mode. Provider compliance doesn’t catch business logic errors. A confidence_score field typed as float can still receive 999.0. Your validator is the right place for range checks, cross-field rules, and semantic constraints.
• Build provider fallback chains for critical paths. When one provider’s API is degraded, your schema should be portable enough to route to a backup. An LLM model router can handle the dispatch logic; libraries like Instructor or Vercel AI SDK make the schema portability practical.
• Track parse failures and business validation failures separately. “Schema compliance rate” and “business validation rate” are different signals. A dashboard that conflates them will mask prompt regressions.
• Test schemas against model updates in CI. Providers update base models without always bumping the model ID in your code. Your CI/CD pipeline for AI agent testing should include schema adherence tests to catch regressions before they hit production.

Choosing the Right Approach for Your Stack

A quick decision guide:

Stop Treating JSON Parse Failures as Acceptable Losses

A 15% failure rate is not a quirk of working with LLMs — it’s a cost you’re paying unnecessarily. The infrastructure to eliminate it exists across all three major providers, the migration path is well-defined, and the schema design patterns that preserve model quality are learnable in an afternoon.

Pick the highest-volume endpoint in your system that parses LLM output today. Apply native LLM structured outputs with strict mode. Validate downstream with Pydantic or Zod. The reliability difference shows up in your logs immediately — and your retry budget thanks you.