What is the JSON structure of a RAG document?

A RAG document (LangChain/LlamaIndex style) is an object with page_content — the text to be embedded and retrieved — and metadata, a dictionary of attributes like source, page, and ids that travel with the text. See the Document example for the full shape and a LlamaIndex TextNode variant.

How is a chunk different from a document?

A document is a whole source file; a chunk is a small, overlapping slice of it (typically 200–500 tokens) that's actually embedded and retrieved. Each chunk inherits the document's metadata plus its own position. Compare the chunk example with the document example.

What does a vector search response look like?

It's typically a matches array, each entry having an id, a similarity score, and the stored metadata. You request results with topK and an optional metadata filter. The vector search example shows the query and response side by side.

How should a RAG answer cite its sources?

Return the answer plus a citations array carrying each source's id, location, and ideally the exact supporting quote, so users can verify the claim. The RAG citation example shows inline markers and the grounded no-answer case. For the broader picture, read JSON for AI & LLM Engineering.

RAG JSON Formats Explained: Documents, Chunks, Vectors & Citations

Retrieval-augmented generation (RAG) is how you give a language model knowledge it wasn't trained on: you retrieve relevant text from your own data and put it in the prompt. Architecturally, RAG is a pipeline of JSON transformations — a document becomes chunks, chunks become vectors, a query becomes a search, matches become context, and the model returns a cited answer.

If you can picture the JSON at each stage, the whole system stops being magic. This guide walks the pipeline end to end, and every stage links to a copy-ready example you can open, format, validate, or turn into TypeScript types.

> All of these live in the AI & LLM category of the JSON Examples Library.

The RAG pipeline at a glance

text

Document  →  Chunks  →  Embeddings  →  Vector store
                                            │
User query  →  Embedding  →  Search (topK + filter)  →  Matches
                                                           │
                            Matches + query  →  LLM  →  Cited answer

Each arrow is a JSON payload. Let's take them in order.

1. Loading: the Document

Ingestion starts by loading source files into a uniform object. In LangChain and LlamaIndex this is a Document: the text plus metadata that travels with it through the whole pipeline.

LangChain / LlamaIndex Document — page_content (the text) plus metadata (source, page, ids). The metadata is what later powers citations and filtering, so set it carefully at load time.

2. Chunking: splitting documents into retrievable pieces

Whole documents are too large to embed or fit in context, so they're split into overlapping chunks. Each chunk carries its parent's metadata plus its own position.

RAG Document Chunk — chunk text, a score once retrieved, and metadata like source, chunkIndex, and tokenCount.

A common sweet spot is 200–500 tokens per chunk with 10–20% overlap so context isn't lost at boundaries. Smaller chunks improve precision; larger chunks preserve more context per hit.

3. Embedding: turning text into vectors

Every chunk (and later, every query) is converted to a dense float vector by an embedding model. Similar meanings land close together in vector space.

Embedding Vector Response — the data[].embedding array plus the model and token usage. Real vectors have 384–3072 dimensions; the example truncates them for readability.

Two rules: embeddings are only comparable within the same model, and if you change models you must re-embed your whole corpus.

4. Storing & querying: the vector database

Vectors and their metadata go into a vector database (Pinecone, Qdrant, Weaviate, pgvector). At query time you embed the user's question and ask for the nearest neighbours.

Vector Database Search Query — a topK query with a metadata filter and the ranked matches response (each with an id, a similarity score, and metadata).

Metadata filtering is what makes RAG production-ready: it enables multi-tenant isolation, recency filters, and permission-aware retrieval. Always check which similarity metric your index uses (cosine, dot product, Euclidean) before you threshold on score.

5. Generating: the cited answer

Finally, the top matches plus the question go to the LLM, which writes an answer grounded in the retrieved text. A good RAG response returns the sources it used.

RAG Answer with Citations — the answer, an array of citations (with the exact supporting quote and source), and a confidence score. It also shows the all-important no-answer shape.

The single most important behaviour: when retrieval finds nothing relevant, a reliable system abstains — it returns a grounded refusal with empty citations rather than hallucinating. That one branch separates trustworthy RAG from confident nonsense.

6. Carrying the conversation

In a chat RAG app, retrieval happens turn by turn, so you also manage conversation state.

Chat Conversation History — the messages array, including summarized memory to keep long threads under the context window.

Debugging RAG with JSON tools

RAG bugs are almost always retrieval bugs, and they're visible in the JSON:

Inspect retrieved matches. Drop the vector-store response into the JSON Tree Explorer to see exactly which chunks came back and their scores — wrong or low-score matches mean your chunking or embedding needs work, not your prompt.
Validate chunk metadata. Inconsistent metadata breaks citations and filters. Run a sample through the JSON Validator.
Type your payloads. Paste the chunk or match shape into JSON to TypeScript so your pipeline code is type-safe end to end.