What is prompt caching?

Prompt caching stores the processed form of a stable prompt prefix so subsequent calls that begin with the same prefix reuse it instead of reprocessing it. Providers bill cached input at a large discount — up to around 90% off — which cuts cost and latency for repeated context.

How should I structure a prompt to maximize cache hits?

Put stable content first and volatile content last: system prompt, then tool definitions, then static documents, and finally the conversation history and the current user input. The cache matches from the start of the prompt to the first token that differs, so keeping changes at the end maximizes the cached prefix.

Why does JSON key order affect prompt caching?

Caching compares exact tokens, not meaning. Two JSON objects with the same data but different key order serialize to different bytes and produce a cache miss. Serialize the stable prefix once and reuse that exact string, or use deterministic key ordering, so the prefix stays byte-identical across calls.

What content is worth caching?

Large, stable, reused context: big system prompts, tool schemas, long documents you ask many questions about, and multi-turn histories. It's not worth it for one-off prompts with no repeated prefix or when the stable portion is very small.

How do I tell if my caching is working?

Check your provider's cache-hit metrics in the usage response, and diff two consecutive prompts — the first difference should be the new user input and nothing earlier. A difference in the system or tool region means volatile data leaked into the prefix and is breaking the cache.

Prompt Caching: Structure Your JSON to Cut LLM Costs up to 90%

The cheapest tokens are the ones you don't reprocess

Every LLM call re-reads its entire prompt from scratch — and if that prompt includes the same 3,000-token system message, tool schema, and knowledge base on every request, you're paying full price to process identical text over and over. Prompt caching fixes this: providers store the processed form of a stable prompt prefix and, on the next call that starts with the same prefix, reuse it at a large discount. Anthropic advertises up to ~90% savings on cached input, Google 75–90%, OpenAI ~50%.

The catch is that caching keys on an exact prefix match, which turns cost optimization into a JSON-ordering problem. This is the concrete, mechanical companion to the token-cost strategies and context engineering — get the layout right and the discount is nearly free.

How prefix caching works

The cache matches from the start of the prompt up to the first token that differs. Everything before the first difference is a cache hit; everything from the first difference onward is processed fresh. Picture two calls:

text

Call 1:  [ system + tools + docs ]  [ user question A ]
Call 2:  [ system + tools + docs ]  [ user question B ]
              ↑ identical prefix          ↑ differs
           CACHE HIT (cheap)          processed fresh

The implication is a single rule that governs everything else: stable content first, volatile content last.

The ordering rule in practice

Structure the prompt so the parts that never change sit at the front, and the parts that change every call sit at the back:

System prompt — rules, role, output contract. Rarely changes.
Tool / function definitions — the JSON Schemas for your tools. Stable across calls.
Static knowledge — documents, examples, policies reused across requests.
— cache boundary —
Conversation history — grows, but the earlier turns are a stable prefix.
Current user input — different every call.

Put the tool schema *after* the user's message and you've poisoned the cache: the volatile message now sits in the prefix, so nothing after it can be reused. The same static bytes, reordered, is the difference between a 90% discount and full price.

The killer detail: JSON key order must be byte-stable

This is where JSON specifically bites. Caching compares exact tokens, so a prompt that's *semantically* identical but *serialized* differently is a cache miss. If you build your system context or tool definitions by serializing objects, and your serializer doesn't preserve key order, you can silently break caching:

text

Call 1:  {"role":"admin","limits":{"rpm":60}}
Call 2:  {"limits":{"rpm":60},"role":"admin"}   ← same data, different bytes → MISS

Rules that keep the prefix byte-identical:

Serialize the stable prefix once and reuse the exact string — don't re-serialize per call.
Use deterministic serialization (stable key order, consistent whitespace) if you must rebuild it. JSON.stringify preserves insertion order, so build the object consistently; sort keys if any source is non-deterministic.
Don't inject per-call values into the stable region — a timestamp, a request ID, or a reordered key anywhere in the prefix invalidates everything after it.

What's worth caching

Caching has a small write cost and a time-to-live, so it pays off when a substantial, stable chunk is reused within the window:

Large system prompts and tool schemas reused across every request — the classic win.
Long documents you ask many questions about (RAG context, a codebase, a contract).
Multi-turn conversations, where each turn re-sends the growing history as a stable prefix.

It's less useful for one-off calls with no repeated prefix, or when the stable portion is tiny.

Measuring the payoff

Check the size of your stable prefix with a JSON size count — the bigger the cached region, the bigger the savings.
Watch cache-hit metrics in your provider's usage response; a low hit rate usually means something volatile crept into the prefix.
Diff two consecutive prompts with a JSON diff — the first difference should be exactly the user's new input, nothing earlier. If the diff shows a change up in the system or tools region, that's your cache leak.

Prompt caching is one of the highest-ROI changes in an LLM app: no quality trade-off, often a one-time reordering, and a large recurring cost cut. For the broader picture see reducing token cost with JSON, semantic caching, and context engineering.