Syntactic validity is not enough
A model can return perfectly valid JSON that is completely wrong for your application: missing required fields, a string where you need a number, an enum value you never defined. JSON.parse() will happily accept all of it. The only way to catch these is schema validation.
The validation pipeline
A production-grade LLM data pipeline has three gates:
- Parse — is it valid JSON syntax?
- Validate — does it match the expected schema (fields, types, enums, ranges)?
- Sanity-check — are the values plausible (dates not in the year 9999, totals not negative)?
Defining a schema once
Write the schema in one place and use it both to constrain the model and to validate the response. Example JSON Schema:
{
"type": "object",
"properties": {
"ticket_id": {"type": "string", "pattern": "^TK-[0-9]{6}$"},
"priority": {"type": "string", "enum": ["low", "medium", "high", "urgent"]},
"tags": {"type": "array", "items": {"type": "string"}, "maxItems": 5},
"due_date": {"type": "string", "format": "date"}
},
"required": ["ticket_id", "priority"],
"additionalProperties": false
}Validating with Ajv (JavaScript)
import Ajv from "ajv";
import addFormats from "ajv-formats";
const ajv = new Ajv({ allErrors: true });
addFormats(ajv);
const validate = ajv.compile(schema);
function validateModelOutput(raw: string) {
const data = JSON.parse(raw);
if (!validate(data)) {
throw new Error("Schema validation failed: " + ajv.errorsText(validate.errors));
}
return data;
}Validating with Pydantic (Python)
from pydantic import BaseModel, field_validator
from typing import Literal
class Ticket(BaseModel):
ticket_id: str
priority: Literal["low", "medium", "high", "urgent"]
tags: list[str] = []
@field_validator("ticket_id")
@classmethod
def check_id(cls, v: str) -> str:
if not v.startswith("TK-"):
raise ValueError("ticket_id must start with TK-")
return v
ticket = Ticket.model_validate_json(raw) # parses AND validatesValidating with Zod (TypeScript)
import { z } from "zod";
const Ticket = z.object({
ticket_id: z.string().regex(/^TK-\d{6}$/),
priority: z.enum(["low", "medium", "high", "urgent"]),
tags: z.array(z.string()).max(5),
});
type Ticket = z.infer<typeof Ticket>;
const ticket = Ticket.parse(JSON.parse(raw)); // typed + validatedThe retry-on-failure pattern
When validation fails, the best recovery is often to send the error back to the model and ask it to fix its own output:
Your previous JSON failed validation with this error:
"priority" must be one of: low, medium, high, urgent
Return corrected JSON only.This "validate then re-prompt with the error" loop pushes reliability toward 100% without human intervention. Cap it at two or three attempts to avoid runaway cost.
Best practices
- Set
additionalProperties: falseso unexpected keys are caught, not silently ignored. - Validate ranges and formats, not just types (
minimum,maxLength,format). - Log the raw response whenever validation fails — it is your best prompt-tuning signal.
- Keep the same schema in version control; treat it as an API contract.