Context Engineering: Designing the JSON an LLM Sees Every Call

Why prompt engineering wasn't enough

Prompt engineering optimized a single instruction to a chatbot. Agents broke that model: they loop, call tools, remember, and run for many turns, and their failures aren't "the prompt was worded badly" — they're state-management failures. The wrong document got retrieved, the history overflowed the window, a tool result was formatted so the model misread it. The 2026 discipline that replaced prompt engineering is context engineering: the deliberate design of *everything* the model sees on every inference call.

And when you look at what's actually in that context window, it's structured data being assembled and serialized — which makes context engineering, in large part, a JSON structuring problem. This extends structuring RAG context and chat message formats from a single call to the whole agent loop.

What's in the context window

Every call to the model is a freshly-assembled payload. The context engineer decides what goes in it and in what shape:

• System prompt — role, rules, output contract.
• Tool definitions — the JSON Schema for each tool the model may call.
• Retrieved documents — RAG results, formatted for the model to cite.
• Conversation history — prior turns, often pruned or summarized.
• Long-term memory — facts recalled from a store about the user or task.
• Current input — the user's latest message or the loop's next step.

Each of these is data your code fetches, filters, formats, and concatenates before serialization. Get the assembly right and the model performs; get it wrong — stale memory, an unbounded history, a tool result buried mid-prompt — and it fails in ways no prompt tweak fixes.

The three kinds of memory

A key insight of context engineering is that "memory" isn't one thing — production agents use three, each with a different structure:

1. Short-term (working) memory — the recent conversation and last few steps, held in the window itself. Bounded by the token budget, so it must be pruned or summarized as it grows.
2. Long-term memory — a vector database of embeddings for semantic recall ("what did the user tell me last week?"). Retrieved by similarity, then injected as JSON. See AI agent memory in JSON and vector DB metadata.
3. Structured memory — a relational or key-value store of *facts*: the user's preferences, account state, policies, current order. Queried precisely and injected as clean JSON objects.

The engineering is deciding, on each turn, *which slice* of each memory to pull into the finite window — and formatting it so the model can use it.

The core constraint: a finite token budget

The context window is fixed and every token costs money and latency, so context engineering is fundamentally budget allocation. You can't inject everything, so you manage the budget with a few techniques:

• Selective retrieval — RAG that pulls only the most relevant chunks, not the whole knowledge base.
• Summarization / compaction — replacing a long history with a compact summary once it grows past a threshold.
• Compression — tools that shrink verbose context while preserving meaning.
• Pruning — dropping stale or low-value entries rather than letting them accumulate.

The goal is the highest *signal density* per token: everything the model needs to act correctly, and as little else as possible.

Why the JSON structure matters, not just the content

The *shape* of what you inject changes how well the model uses it. Practical structuring rules that pay off:

• Format tool results as clean, minimal JSON, not raw dumps. A 40-field API response where the model needs 3 fields wastes budget and buries the signal — filter before injecting.
• Keep keys stable and descriptive. {"status": "shipped"} is easier for the model to reason over than a positional array, and stable keys help caching (below).
• Delimit the sections clearly so the model can tell system rules from retrieved facts from history.
• Order deliberately — put stable, reusable context (system, tools, schema) *first* and volatile context (the current turn) *last*. This isn't only for readability; it's what makes prompt caching effective and cuts cost dramatically.
• Right-size verbosity. Sometimes compact JSON is best; sometimes a token-lean format like TOON saves budget on large tabular context.

Building and debugging context

Treat the assembled context as an artifact you can inspect, because that's where agents fail:

1. Log the full serialized context for each call — this is LLM observability — then format it to read what the model actually saw.
2. Diff two turns' context with a JSON diff to catch when memory or retrieval injected something wrong or dropped something needed.
3. Measure the budget. Check the size of each section so you know what's consuming the window before you hit the limit.

The mental shift is this: you're not writing a prompt, you're engineering a data pipeline whose output is the JSON the model reasons over. For the pieces that feed it, see RAG context structuring, agent memory in JSON, and multi-agent state.