What is memory in an AI system?
Memory is anything a system stores outside a single request so that a later request can use it. The model has no memory of its own; the harness does. In practice the word covers three different mechanisms that are constantly confused: the state inside the current request, a durable store the harness writes to and reads from, and consolidated notes derived from earlier sessions.
The three things people call memory
The first is in-request state: the conversation and tool results the harness has assembled for this call. It feels like memory to a user and it is not durable at all. When the process ends, or when the harness prunes it to fit the context window, it is gone.
The second is a durable store. A file, a table, a vector index, a key-value store: somewhere the harness writes facts and later reads them back. This is the only kind that survives a restart, and it is ordinary software with ordinary problems, namely who writes to it, who can read it, and what happens when two entries disagree.
The third is consolidated memory: notes derived from past sessions rather than copied from them. Instead of storing every transcript, a separate pass reads what happened and writes down what turned out to matter. The Generative Agents work formalised this as a memory stream plus periodic reflection, where higher-level observations are synthesised from raw ones and are what gets retrieved later. Products describe the same idea under different names, and it is the kind most likely to be oversold, because the quality of the notes matters far more than the fact that notes exist.
Memory is not the context window
The context window is a ceiling on one request. Memory is state between requests. They interact in exactly one place: memory only affects a model's behaviour when the harness reads something out of it and puts it into a request, spending window budget to do so.
This is why a bigger window is not a substitute for memory, and memory is not a substitute for a bigger window. If you need last month's decision, no window size will help unless something stored it. If you need forty documents in one call, no memory system helps unless they fit.
Memory vs retrieval: is RAG memory?
Mechanically they are the same move: fetch something from outside the request and put it in. The useful distinction is where the content came from. Retrieval usually serves a corpus you already had, such as your documentation, and answers the question what does the source say. Memory serves material the system itself produced, such as what a user told it or what an earlier run concluded, and answers the question what happened before.
The reason to keep them apart in a design is trust. A document you indexed is as correct as your source. A memory the system wrote about itself is a claim it made once, and it will be resurfaced later with the same confidence whether or not it was right.
How memory goes wrong
Staleness first: a stored fact has no expiry unless you gave it one, and a preference recorded in March will be applied in September. Contradiction next: nothing stops two entries from disagreeing, and the retrieval step has no notion of which is truer, only which is more similar to the query. Then unbounded growth, where a store that is only ever appended to slowly fills the retrieved slots with noise and crowds out the entries that matter. And finally scope: memory written during one user's session is a data protection question the moment it can surface in another's.
The mitigations are unglamorous and effective. Timestamp everything and prefer recent entries. Write few, high-value entries rather than many. Let the system delete and correct its own notes, not only add to them. Partition the store by whoever it belongs to, and be able to answer what is stored about a given person.
Common questions
Memory in AI systems: frequently asked
Does the model remember our previous conversation?
No. Each request is evaluated on its own, and nothing is retained on the provider's side between calls. If a previous conversation appears to be remembered, the program around the model stored it and sent it again as part of the new request. That is why the same model can seem to have a long memory in one product and none at all in another.
Is memory the same as the context window?
No. The context window is a per-request limit on how much the model can consider at once. Memory is state kept outside the request between calls. They only meet when the harness reads from memory and spends part of the window on what it found, which means memory always costs context to use.
Should an agent be allowed to write its own memory?
It is useful and it needs limits. An agent that records what worked gets better across sessions, but it will also record mistaken conclusions with the same confidence and retrieve them later as fact. Timestamp entries, keep them few and specific, allow correction and deletion rather than append only, and treat anything the system wrote about itself as a claim rather than a source.
What is memory consolidation?
It is a separate pass that reads what happened in earlier sessions and writes down the parts worth keeping, instead of storing whole transcripts. The Generative Agents research described this as reflection over a memory stream, synthesising higher-level observations that are then what gets retrieved. The value depends entirely on the quality of those notes, so the pass that writes them deserves as much attention as the retrieval that reads them.
Sources
Where these facts come from
- Generative Agents: Interactive Simulacra of Human Behavior: the memory stream and reflection design that most consolidation schemes descend from
- Anthropic docs: context windows: why stored state only matters once it is placed back into a request
- Model Context Protocol specification: resources and tools as the interface a harness uses to reach a durable store