How to Get Reliable JSON Output from LLMs (ChatGPT, Claude, Gemini)

Why LLM JSON is unreliable by default

Large language models are trained to produce natural language, not strict data formats. Ask a model to "return JSON" and you will eventually get a response wrapped in Markdown fences, prefixed with "Sure! Here is your JSON:", containing trailing commas, or with a stray sentence after the closing brace. Any of these will make JSON.parse() throw.

If your application feeds model output straight into code, a single malformed response can crash a request or corrupt a pipeline. The good news: with the right combination of API features, prompting, and validation you can get valid JSON close to 100% of the time. This guide covers every layer.

Layer 1: Use the provider's structured output feature

The single biggest win is to stop asking nicely and start using the API's native JSON support. All three major providers now constrain the decoder so the output is guaranteed to be syntactically valid JSON.

With OpenAI Structured Outputs you pass a JSON Schema and the model is constrained to it:

from openai import OpenAI
client = OpenAI()

schema = {
    "type": "object",
    "properties": {
        "sentiment": {"type": "string", "enum": ["positive", "negative", "neutral"]},
        "score": {"type": "number"},
        "keywords": {"type": "array", "items": {"type": "string"}},
    },
    "required": ["sentiment", "score", "keywords"],
    "additionalProperties": False,
}

resp = client.chat.completions.create(
    model="gpt-4.1",
    messages=[{"role": "user", "content": "Analyze: 'The checkout flow is fast but the search is broken.'"}],
    response_format={
        "type": "json_schema",
        "json_schema": {"name": "review", "schema": schema, "strict": True},
    },
)
data = resp.choices[0].message.content  # guaranteed valid JSON string

For Claude, the idiomatic pattern is to define a tool and let the model "call" it — the arguments come back as schema-valid JSON:

import anthropic
client = anthropic.Anthropic()

msg = client.messages.create(
    model="claude-opus-4-8",
    max_tokens=1024,
    tools=[{
        "name": "record_review",
        "description": "Record a structured product review analysis.",
        "input_schema": {
            "type": "object",
            "properties": {
                "sentiment": {"type": "string", "enum": ["positive", "negative", "neutral"]},
                "score": {"type": "number"},
                "keywords": {"type": "array", "items": {"type": "string"}},
            },
            "required": ["sentiment", "score", "keywords"],
        },
    }],
    tool_choice={"type": "tool", "name": "record_review"},
    messages=[{"role": "user", "content": "Analyze: 'Fast checkout, broken search.'"}],
)
# msg.content[0].input is already a parsed dict matching the schema

Layer 2: Prompt for JSON even when you cannot use schemas

Sometimes you are stuck with a model or endpoint that has no structured-output mode. Prompting still matters. The rules that move the needle most:

• Show the exact shape. Include a literal example of the JSON you want, not a description of it.
• Say "Respond with only JSON. No Markdown, no commentary." Models follow explicit negative instructions surprisingly well.
• Name the fields and types. "Return an object with title (string), tags (array of strings), and published (boolean)."
• Prefill the opening brace. With APIs that support an assistant prefix, start the assistant turn with { so the model cannot add a preamble.

A compact, effective prompt:

Extract the event details. Respond with ONLY a JSON object, no Markdown.

Schema:
{
  "title": string,
  "date": string (ISO 8601),
  "location": string | null,
  "attendees": number
}

Text: "Team sync on March 3rd 2026, 14 people, in the Ahmedabad office."

Layer 3: Always validate and repair

Treat model output as untrusted input — because it is. Even with structured outputs, network glitches and truncation happen. A robust client does three things:

1. Strip fences. Remove a leading ```json` and trailing `````` if present.
2. Parse defensively. Wrap JSON.parse in try/catch and attempt a repair pass on failure (fix trailing commas, single quotes, unterminated strings).
3. Validate against a schema. Parsing only proves it is syntactically valid JSON, not that it has the fields you need.

import { z } from "zod";

const Review = z.object({
  sentiment: z.enum(["positive", "negative", "neutral"]),
  score: z.number(),
  keywords: z.array(z.string()),
});

function parseModelJson(raw: string) {
  const cleaned = raw.trim().replace(/^\`\`\`json\n?|\`\`\`$/g, "");
  const obj = JSON.parse(cleaned);     // may throw — handle upstream
  return Review.parse(obj);            // throws if fields are wrong
}

If a response still will not parse, paste it into a JSON repair tool to see exactly where it breaks — a missing brace or an unterminated string is obvious once highlighted.

A reliability checklist

• Use the provider's JSON Schema / tool-use mode whenever available.
• Keep schemas small and flat; deeply nested schemas are harder for models to satisfy.
• Set additionalProperties: false so the model cannot invent fields.
• Lower the temperature (0–0.3) for extraction tasks.
• Validate every response with a schema library (Zod, Pydantic, Ajv).
• Log raw responses that fail so you can tune the prompt.