Should I send retrieved context as JSON or plain text?

JSON when you need per-chunk metadata, scores, and citations — which is almost always. Minify it to keep token costs down.

How do I stop the model answering from outside the context?

Instruct it to use only the provided context, ask it to return a grounded flag and citations, and reject answers with no citations.

How many chunks should I include?

Enough to cover the answer, rarely more than 5-10. Rank by similarity, trim to your token budget, and prefer precision over volume.

Structuring RAG Context as JSON: Chunks, Metadata & Citations

Why the shape of your context matters

In Retrieval-Augmented Generation (RAG), you fetch relevant documents and inject them into the prompt so the model answers from your data instead of its training. Answer quality depends not just on *what* you retrieve but on *how you structure it*. Hand the model a wall of concatenated text and it loses track of sources; hand it well-shaped JSON and it can cite, rank, and reason over chunks reliably.

A chunk schema worth copying

Each retrieved passage should carry its text plus the metadata the model (and your app) needs:

json

{
  "id": "doc_42#chunk_3",
  "text": "Refunds are processed within 5-7 business days after approval.",
  "source": "Refund Policy",
  "url": "https://example.com/policy/refunds",
  "score": 0.87,
  "updated": "2026-04-01"
}

id — a stable identifier so the model can cite a specific chunk.
text — the passage itself, already trimmed to a sensible size.
source / url — for attribution and clickable citations.
score — the retrieval similarity, so the model can weight stronger matches.
updated — lets the model prefer fresher information.

Assemble the prompt as structured context

Pass an array of chunks, minified, inside a clearly delimited block:

text

Answer the question using ONLY the context below. Cite sources by id.
If the answer is not in the context, say "I don't know".

<context>
[{"id":"doc_42#chunk_3","text":"Refunds are processed within 5-7 business days...","source":"Refund Policy"}]
</context>

Question: How long do refunds take?

Minify the JSON to save tokens, and fence it with tags so the model treats it as data, not instructions.

Ask for structured, cited answers

Have the model return JSON too, so your UI can render citations and detect ungrounded answers:

json

{
  "answer": "Refunds take 5-7 business days after approval.",
  "citations": ["doc_42#chunk_3"],
  "grounded": true
}

If grounded is false or citations is empty, you can suppress the answer or fall back — a simple, powerful guardrail against hallucination.

Chunking tips that improve retrieval

Right-size chunks. 200-500 tokens is a common sweet spot — small enough to be specific, large enough to carry context.
Overlap slightly. A 10-20% overlap prevents answers from being split across a boundary.
Keep metadata with the chunk. Store source, title, and timestamps alongside the vector so they return with every match.
Deduplicate. Near-identical chunks waste context budget; collapse them before sending.

Rank and trim before the prompt

Retrieval often returns more than you can afford to send. Before building the prompt: sort by score, drop anything below a relevance threshold, keep the top N that fit your token budget, and optionally re-rank with a cross-encoder for precision. A focused five-chunk context usually beats a noisy twenty-chunk one — in both answer quality and cost.