Why agents need memory
A stateless LLM forgets everything between calls. To build an assistant that remembers a user's preferences, an agent that tracks progress on a long task, or a chatbot that recalls earlier in the conversation, you need memory — and memory is data you store, retrieve, and inject back into the prompt. JSON is the natural format for it.
Two kinds of memory
| Type | Holds | Lifespan | Storage |
|---|---|---|---|
| Short-term | The current conversation | One session | The message array / a state object |
| Long-term | Durable facts about the user/world | Across sessions | A database or vector store |
A capable agent uses both: short-term for "what we just said," long-term for "what I know about you."
Short-term memory as a state object
Beyond the raw message history, agents often keep a compact JSON state of what matters right now:
{
"user_goal": "Plan a 5-day trip to Japan",
"constraints": { "budget_usd": 2000, "month": "October" },
"decided": ["Fly into Tokyo", "3 nights Kyoto"],
"open_questions": ["Rail pass or flights?"]
}Updating and re-injecting this small object each turn keeps the agent on track far better than hoping it re-reads a long transcript.
Long-term memory: facts and summaries
For durable memory, extract structured facts and store them so they can be retrieved later:
{
"user_id": "usr_91",
"memories": [
{ "fact": "Prefers window seats", "type": "preference", "created": "2026-06-01" },
{ "fact": "Allergic to peanuts", "type": "constraint", "confidence": 0.98 }
]
}Two common strategies:
- Summarization. Periodically compress old turns into a short JSON summary, so the agent keeps the gist without the token cost of the full history.
- Vector retrieval. Embed each memory and store it in a vector database; at each turn, retrieve the few most relevant memories and inject them. This scales to thousands of facts without bloating the prompt.
The memory loop
- Retrieve relevant long-term memories (by recency, type, or similarity).
- Assemble the prompt: system message + retrieved memories + short-term state + recent turns.
- Respond with the model.
- Extract new facts worth keeping and write them back to long-term storage.
Each step passes JSON, which makes the whole pipeline inspectable and testable.
Pitfalls to avoid
- Unbounded growth. Cap and prune memories; deduplicate near-identical facts.
- Stale or contradictory facts. Store timestamps and confidence; prefer recent, high-confidence memories and let new facts supersede old ones.
- Over-injection. Retrieving twenty memories per turn wastes tokens and distracts the model. Retrieve the few that matter.
- Privacy. Memory is user data — store it securely, scope it per user, and let users view and delete it.