Batch LLM Requests with JSONL: OpenAI and Anthropic Batch APIs

July 1, 2026·8 min read·AI & JSON

Half the price, if you can wait

Not every LLM job is interactive. Classifying a million reviews, generating embeddings for a catalog, backfilling summaries — these don't need a response in 200ms, they need to finish overnight cheaply. That's what Batch APIs are for: submit a big pile of requests, get results within a window (often up to 24h), and pay roughly 50% less than the synchronous API. Both OpenAI and Anthropic offer them, and both speak JSONL — JSON Lines, one self-contained request per line.

The input file: one request per line

A batch input file is JSONL where each line is a complete request wrapped with a `custom_id` you choose. That id is the contract that lets you match results back to inputs — because batch results come back out of order.

text

{"custom_id":"review-001","method":"POST","url":"/v1/chat/completions","body":{"model":"gpt-5.2","messages":[{"role":"user","content":"Sentiment of: Great value!"}]}}
{"custom_id":"review-002","method":"POST","url":"/v1/chat/completions","body":{"model":"gpt-5.2","messages":[{"role":"user","content":"Sentiment of: Broke in a week."}]}}

Each line is independent, which is the whole point of JSONL: you can generate the file by streaming records without ever holding the full array in memory, and the provider can process lines in parallel.

Why JSONL and not a JSON array

A giant [ ... ] array must be fully parsed before the first item is usable and fully built in memory before it's written. JSONL is append-friendly and streamable — write a line, flush, repeat — which is exactly what you want for hundreds of thousands of requests. It's the same reason JSONL is the default for fine-tuning datasets and log streams.

Reconciling results

The output is also JSONL, one result per line, each echoing your custom_id:

text

{"custom_id":"review-002","response":{"status_code":200,"body":{"choices":[{"message":{"content":"negative"}}]}}}
{"custom_id":"review-001","response":{"status_code":200,"body":{"choices":[{"message":{"content":"positive"}}]}}}

Note the order flipped. Your reconciliation code must key on `custom_id`, never on line position. Also check each line's status — a batch can partially fail, and individual lines carry their own error objects.

Building and checking batch files

Malformed JSONL is the most common batch failure — one bad line rejects the file. Validate before you upload:

Generate the file, then spot-check lines by pasting them into the JSON Formatter — each line must be valid JSON on its own.
Use the JSON Lines tool to convert between a JSON array and NDJSON while building or debugging the file.
Confirm every custom_id is unique — duplicates make results ambiguous.
Need realistic test requests? Generate a batch of records in the Random JSON Generator and shape them into request lines.

For streaming responses (the opposite end of the latency spectrum) see Streaming Partial JSON from AI, and for cutting per-request tokens see TOON vs JSON.

Frequently asked questions

JSONL (JSON Lines) — one complete request per line for input, one result per line for output. Each line is independent, which makes the file streamable and parallelizable.

Because batch results return out of order and possibly partially failed. The custom_id you assign each request is echoed on its result, so you reconcile by id — never by line position.

Typically around 50% of the synchronous price, in exchange for asynchronous processing within a completion window (often up to 24 hours). Ideal for classification, embeddings, and bulk generation that isn't latency-sensitive.

A malformed line — JSONL requires every line to be valid JSON on its own, so a single stray comma or unquoted key fails the upload. Validate each line before submitting, and ensure custom_id values are unique.

Try JSON Lines / NDJSON

Convert between a JSON array and JSONL while building your batch file.

Open JSON Lines →Formatter →Validator →