Structured Output with LLMs: JSON Schema, Validation, and Error-Handling Patterns

Overview

Structured output is the practice of asking an LLM to return data in a predictable machine-readable shape instead of free-form prose. It is useful for classification, extraction, content pipelines, user interfaces, agent workflows, and any AI app development task where another function will consume the response.

A dependable implementation has several layers:

1. Define the contract. Describe the fields, types, required values, and allowed variations in JSON Schema or an equivalent type definition.
2. Constrain the generation. Use a provider's native structured-output or schema-constrained feature when it fits your model and schema.
3. Validate at the application boundary. Treat the model response as untrusted input, even when the API claims to enforce a format.
4. Handle failure explicitly. Distinguish refusal, truncation, invalid JSON, schema mismatch, timeout, and unavailable-provider errors.
5. Test representative cases. Include empty inputs, ambiguous requests, long content, unusual characters, and expected refusal or fallback scenarios.

These layers solve different problems. JSON mode may help produce syntactically valid JSON, but syntax alone does not guarantee the required fields or correct data types. A schema can describe the expected shape, but validation still belongs in your application. Post-processing can recover from some failures, but it should not silently change meaning.

For broader implementation context, compare structured output with tool calls in this function calling tutorial, and review the AI developer tools comparison before selecting an SDK or provider.

Checklist by scenario

Scenario 1: Extract fields from text

For invoices, support tickets, articles, or forms, start with a narrow object rather than asking the model to reproduce the entire source. Make uncertainty explicit so the model does not have to invent a value.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "additionalProperties": false,
  "properties": {
    "title": { "type": ["string", "null"] },
    "category": {
      "type": "string",
      "enum": ["billing", "technical", "account", "other"]
    },
    "confidence": {
      "type": "string",
      "enum": ["high", "medium", "low"]
    },
    "evidence": { "type": "array", "items": { "type": "string" } }
  },
  "required": ["title", "category", "confidence", "evidence"]
}

Use null for an absent value when that is different from an empty string. Use an enum when downstream code supports a fixed set of states. If an extracted field must be traceable, require evidence or source spans rather than relying on a confidence label alone.

Scenario 2: Generate content metadata

For an AI content or SEO workflow, separate editorial fields from generated prose. A practical response might include a proposed title, description, keywords, and review notes. Keep fields that require human judgment visible instead of presenting them as verified facts.

• Require strings for title and description, with application-side length checks.
• Use an array of strings for keywords and remove duplicates after validation.
• Include a needs_review boolean or review reason for uncertain outputs.
• Store the original input, schema version, model identifier, and validation result with the output.

The AI content and SEO workflow explains where human review fits into a publishing pipeline. Structured output makes that review easier to route, but it does not replace editorial verification.

Scenario 3: Return a decision for an application

For routing, moderation, or customer-support triage, define a small response that a program can act on safely. Prefer a status enum and a separate explanation over a single free-form recommendation.

{
  "decision": "escalate",
  "reason_codes": ["missing_context", "possible_account_issue"],
  "next_action": "request_account_details",
  "user_message": "Please provide the account email so the issue can be reviewed."
}

Validate that next_action is compatible with decision. JSON Schema can check the basic shape; application logic should enforce relationships between fields and authorization rules. Never let an LLM-generated field directly trigger a sensitive operation without independent checks.

Scenario 4: Select an output strategy

Check the current API documentation for the exact feature set before implementation. Provider SDK syntax and supported schema keywords can change independently of your prompt.

What to double-check

Validate syntax and schema separately

First parse the response as JSON. Then validate the resulting value against the intended schema. In pseudocode:

raw = call_model(request)

if raw.status == "refused":
    return safe_fallback("The model did not provide a usable result")

try:
    value = parse_json(raw.text)
except ParseError:
    log_failure("invalid_json", raw.text)
    return retry_or_fallback()

result = validate(schema, value)
if not result.valid:
    log_failure("schema_mismatch", result.errors)
    return retry_or_fallback()

return apply_business_rules(value)

Use a validation library appropriate to your language. Keep error messages available for logs and tests, but avoid returning internal prompts, raw personal data, or provider details to end users.

Check truncation, refusal, and empty results

A response can fail even when the JSON parser is correct. Inspect the API response metadata for completion status, finish reason, refusal indicators, or equivalent fields. A truncated object may be invalid; a short object may be valid but incomplete. Treat both as application-level failures when required fields are absent.

Version the schema

Give each output contract a version, such as ticket_triage.v2. Store it with the prompt and model configuration. If a field changes from a string to an array, or an enum gains a new state, decide whether old consumers can handle the change before deploying it.

Test the full distribution of inputs

Your test set should include normal examples and difficult cases: empty text, conflicting instructions inside retrieved content, multilingual input, very long input, invalid dates, duplicate items, missing identifiers, and text that asks the model to ignore the extraction task. A prompt testing framework can help you compare schema validity, field accuracy, refusal handling, and latency across revisions.

Common mistakes

• Confusing JSON mode with correctness. Parsing success does not prove that the response matches your business contract.
• Making every field required. If the source may not contain a value, require a deliberate null or an uncertainty state instead of encouraging invention.
• Allowing arbitrary properties by accident. Decide whether extra keys are acceptable. Strict objects make downstream behavior more predictable.
• Using a repair prompt as the primary design. Repair is a fallback, not a substitute for a clear schema and constrained generation.
• Retrying without limits. Set a maximum number of attempts, record each failure category, and provide a deterministic fallback.
• Trusting generated instructions. A field such as tool_to_call or send_email must be checked against an allowlist and the current user's permissions.
• Ignoring prompt and schema drift. A small wording change can alter optional fields, enum usage, or extraction behavior. Review both together.

For retrieval-based applications, also inspect whether the model is extracting from the supplied context or filling gaps from its general knowledge. The RAG tutorial and RAG production checklist cover related reliability controls.

When to revisit

Revisit this structured output checklist whenever you add a new output type, change a schema, switch models, update an SDK, modify a system prompt, or connect the response to a new downstream action. It is also worth reviewing before seasonal planning cycles, when input volume or content changes, and whenever workflows or tools change.

Before release, confirm the following:

1. The schema expresses real business requirements, including legitimate unknown values.
2. The selected API feature supports the schema features you use.
3. Parsing, validation, refusal, truncation, timeout, and retry paths are tested.
4. Logs capture schema version, failure category, and safe diagnostic context.
5. Side effects require independent authorization and validation.
6. Representative production-like examples pass after prompt, model, or dependency changes.

Reliable JSON is a system property, not a single prompt trick. Define a narrow contract, constrain generation where practical, validate every response, and make failure visible. That workflow gives your LLM application a safer foundation as its prompts, models, and integrations evolve.