AI & LLM

LLM Evaluation / Benchmark Result JSON Example

A JSON example of an LLM evaluation result — per-test scores, metrics, pass/fail, and aggregate stats. Copy-ready for model evals, regression testing, and LLM observability.

JSON

Generate mock data Open in Formatter

{
  "runId": "eval_2025_05_20_a",
  "model": "gpt-4o",
  "dataset": "support-qa-v2",
  "summary": {
    "total": 200,
    "passed": 184,
    "failed": 16,
    "passRate": 0.92,
    "avgLatencyMs": 1180
  },
  "metrics": {
    "faithfulness": 0.94,
    "answerRelevancy": 0.91,
    "contextPrecision": 0.88
  },
  "cases": [
    {
      "id": "case_017",
      "input": "How do I reset my password?",
      "expected": "Settings → Security → Reset password",
      "output": "Go to Settings, then Security, and click Reset password.",
      "scores": {
        "exactMatch": 0,
        "semanticSimilarity": 0.93
      },
      "passed": true
    }
  ],
  "createdAt": "2025-05-20T12:00:00Z"
}

Field Reference

summaryrequiredobjectAggregate counts and pass rate for the whole run

metricsoptionalobjectScored quality dimensions (faithfulness, relevancy) on a 0–1 scale

casesrequiredarray<object>Per-test-case records with input, output, scores, and pass/fail

cases[].scoresrequiredobjectOne or more metric scores for the case (exact match, similarity, etc.)

cases[].passedrequiredbooleanWhether the case met the threshold for its metrics

Variants

LLM-as-judge caseA case scored by another model acting as a grader, with a rationale.

LLM-as-judge case

Generate mock data Open in Formatter

{
  "id": "case_044",
  "input": "Summarize the refund policy.",
  "output": "Refunds are available within 30 days.",
  "judge": {
    "model": "gpt-4o",
    "score": 4,
    "max": 5,
    "rationale": "Accurate but omits the condition about used items."
  },
  "passed": true
}

Common Use Cases

→Tracking model quality across versions and prompts over time
→Catching regressions in a CI pipeline before shipping prompt changes
→Comparing models on your own dataset rather than public benchmarks

llm evalevaluationbenchmarktestingmetricsobservability

Validate or format this JSON

One click loads this exact example into the tool — no copy-paste needed. Format it, validate it, explore the tree, or generate TypeScript types instantly.

Format Validate Explore tree Generate TypeScript Minify

Frequently Asked Questions

It's using a strong model to grade another model's output against criteria you define, returning a score and rationale. It scales better than human review and handles open-ended answers that exact-match can't, but you should periodically validate the judge against human labels.

Common ones are exact/semantic match for closed answers, and faithfulness, answer relevancy, and context precision for RAG. Always pair quality metrics with operational ones (latency, cost, token usage) so improvements in quality don't silently regress performance.

Related JSON Examples

AI & LLMOpenAI Chat Completion ResponseOpenAI Chat Completions API response with choices, message, finish_reason, and usage.AI & LLMPrompt Template (Reusable)Reusable prompt template with variables, model params, and few-shot examples.AI & LLMAI Agent Run TraceAgent run with goal, reasoning steps, tool calls, observations, and final output.