Evaluating LLM JSON Outputs: Building a Reliable Eval Set

Why you need evals

"It seems to work" is not a strategy. When an LLM produces JSON for your app, you need a repeatable way to measure whether changes to the prompt, model, or schema make things better or worse. That is what an eval set is: a fixed collection of inputs with known-correct outputs that you score automatically.

Without evals, every prompt tweak is a guess. With them, you have a number that tells you if you improved.

Anatomy of an eval case

Each case pairs an input with the expected structured output:

{
  "id": "case_012",
  "input": "Refund my order TK-204512, it arrived broken.",
  "expected": {
    "intent": "refund",
    "order_id": "TK-204512",
    "sentiment": "negative"
  }
}

Collect 50-200 cases that cover the variety you see in production — edge cases, ambiguous inputs, and the failure modes you have already hit.

Three levels of scoring

LLM JSON should be scored at increasing strictness:

A response can pass parse and schema but still get the intent wrong — accuracy scoring catches that.

A simple eval runner

import { z } from "zod";

const Schema = z.object({
  intent: z.enum(["refund", "question", "complaint", "other"]),
  order_id: z.string().nullable(),
  sentiment: z.enum(["positive", "neutral", "negative"]),
});

async function runEvals(cases) {
  let parseOk = 0, schemaOk = 0, fieldHits = 0, fieldTotal = 0;
  for (const c of cases) {
    const raw = await callModel(c.input);
    let obj;
    try { obj = JSON.parse(raw); parseOk++; } catch { continue; }
    const result = Schema.safeParse(obj);
    if (!result.success) continue;
    schemaOk++;
    for (const key of Object.keys(c.expected)) {
      fieldTotal++;
      if (obj[key] === c.expected[key]) fieldHits++;
    }
  }
  return {
    parseRate: parseOk / cases.length,
    schemaRate: schemaOk / cases.length,
    fieldAccuracy: fieldHits / fieldTotal,
  };
}

Now a single number — field accuracy — tells you whether a change helped.

Beyond exact match

Some fields need fuzzier scoring: a free-text summary is not graded by equality. Options include string similarity, checking that key facts appear, or an LLM-as-judge that rates the answer against a rubric. Reserve LLM judging for subjective fields; use exact and schema checks for structured ones because they are cheap, fast, and deterministic.

Make evals part of your workflow

• Version your eval set in git alongside the code.
• Run evals in CI so a prompt change that drops accuracy fails the build.
• Add every production failure as a new eval case so you never regress on the same bug twice.
• Track scores over time to see whether the system is improving.