Long-term memory for agents: beyond the context window

An agent that helps the same user across weeks of sessions faces a problem that neither a bigger context window nor a better retriever solves on its own. It has to decide what, from its own history of interactions, is worth keeping, and then find it again later. That is the memory problem, and it is worth separating cleanly from the two adjacent ideas it is often confused with.

Three different things

Long context is what fits inside a single model call. You pack the prompt with as much history and evidence as the window allows, and the model attends over all of it at once. It is simple and, up to a point, effective, but it is bounded by the window and by the fact that accuracy degrades as the window fills. We covered that trade-off in long context versus RAG.

Classic retrieval-augmented generation (RAG) fetches from a fixed external corpus that the agent did not author: documentation, an encyclopaedia, a case-law archive. The corpus is stable, curated ahead of time, and shared across every user. The agent reads from it but does not write to it. Most of the sources in our directory of datasets are exactly this kind of stable knowledge.

Agent memory is the third thing. Here the store is written by the agent itself, from its own conversations and actions, and it is usually specific to one user or one task. The hard parts are not just retrieval but the write path: deciding what to record, when to merge a new fact with an old one, when to overwrite something that has changed, and when to let a detail go. Memory is a read-write system over the agent’s own past, where classic RAG is read-only over someone else’s corpus.

MemGPT and the operating-system analogy

The system that framed this most influentially is MemGPT (Packer et al., 2023), subtitled “Towards LLMs as Operating Systems”. Its central move is to treat the context window like physical memory (RAM) and an external store like disk, then let the model page information between the two. The window holds a working set: a system prompt, recent messages, and a scratch area. When something needs to persist beyond the window, the model issues explicit function calls to write it to external storage, and when it needs something that has been paged out, it calls to read it back.

The elegant part is that the model manages its own memory. MemGPT gives the model tools to edit its working context, search recall storage, and evict older content when the window pressures its limit, much as an operating system handles interrupts and paging. It is evaluated on document analysis beyond the window and on multi-session chat, where an agent is meant to remember, reflect, and evolve across long-running conversations. The design has since been productised (the Letta project descends from it), and the paging metaphor now underpins much of how people reason about tiered agent memory.

Mem0 and extract-then-store layers

A second family treats memory less like paging and more like a curated set of facts. Mem0 (Chhikara et al., 2025) is representative: rather than moving raw conversation in and out of context, it extracts salient information from each turn and stores it as discrete memories in a vector store.

The interesting design choice is on the write path. When a new candidate memory arrives, an LLM-based controller inspects the most similar existing memories and classifies what to do: ADD a new memory, UPDATE an existing one, DELETE something now contradicted, or NOOP if it adds nothing. That is the consolidation and forgetting logic made explicit, and it is what stops the store from becoming an ever-growing pile of near-duplicate and stale assertions. The paper reports large reductions in token cost and latency against passing full history, because at read time the agent retrieves a handful of compact facts rather than replaying the whole transcript. A graph-backed variant, Mem0-g, stores relationships between entities in a graph database instead of a flat vector index.

A related line, A-MEM (Xu et al., 2025), pushes the organisation idea further, giving each memory structured notes and links to related memories, and updating those links as new memories arrive, so the store evolves into something closer to an interconnected knowledge base than a flat list. The common thread across these systems is that memory is not a log; it is a maintained data structure, and the maintenance is where most of the engineering lives.

What memory is actually for

The practical division of labour is clearer once you see the write path. Use retrieval for knowledge that is stable, shared, and external: the specification of a protocol, a company’s product docs, the text of a statute. That belongs in a curated corpus, versioned and evaluated like any other dataset. Use memory for state that is evolving and specific to the individual: a user’s preferences, decisions taken earlier in a project, the current status of a long task, corrections the user has made that should stick.

The two are complementary, not competing. An agent answering a legal question should retrieve the authority from a stable source and remember that this particular user always wants answers grounded in one jurisdiction. Confusing the roles is a common failure: putting durable reference knowledge into per-user memory means every user rebuilds the same facts and none of it is curated, while trying to hold evolving user state in a static corpus means it goes stale the moment it is written.

The ceiling: what benchmarks show

It is tempting to treat memory as a solved substitute for good retrieval. The evaluations do not support that.

LongMemEval (Wu et al., 2024) tests five distinct abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention, using 500 curated questions embedded in chat histories that can be scaled to arbitrary length. The headline result is sobering: commercial assistants and long-context models show a substantial accuracy drop, on the order of thirty per cent, once information has to be recalled across sustained interactions rather than from a fresh window. The paper decomposes memory into indexing, retrieval, and reading, and shows that careful design at each stage (session decomposition, fact-augmented indexing, time-aware query expansion) recovers a good deal of the loss, which is itself the point: memory quality is an engineering variable, not a free property of having a store.

LoCoMo (Maharana et al., 2024) builds very long multi-session conversations, averaging hundreds of turns, and evaluates question answering, event summarisation, and multi-modal dialogue generation over them. It finds that models struggle with long-range temporal and causal structure, and that while long-context and RAG both help, they remain well short of human performance. Two patterns recur across both benchmarks. First, accuracy degrades as the store grows, because more memories means more chances for the retriever to surface the wrong one. Second, questions that require combining several past facts (multi-hop and temporal reasoning) are markedly harder than single-fact recall, which is exactly the regime where a naive top-k lookup over memories falls down.

That second pattern is the same reasoning-over-retrieval problem that shows up whenever a single query cannot name what it needs, and the mitigations rhyme with those in reasoning-intensive retrieval: decompose the question, retrieve iteratively, and reason over what comes back rather than trusting the first hit.

Evaluating a memory system

Because the write path is where memory systems differ, evaluate the write path, not just recall. A store that never forgets will pass a recall test and fail in production once contradictions accumulate. Worth measuring: does the system correctly update when a fact changes, does it abstain when it genuinely has no memory rather than confabulating, and does accuracy hold as the store grows from a handful of sessions to hundreds. LongMemEval’s ability breakdown is a good template for the axes to test.

If your agent also reads from a fixed corpus, evaluate that leg separately with the usual RAG measures. Purpose-built resources such as open-ragbench and the wider RAG benchmarks category exist for the retrieval side, and the broader discipline of measuring grounding is covered in RAG faithfulness evaluation. Treat the two evaluations as distinct: strong retrieval numbers tell you nothing about whether the agent remembers what the user told it last week, and strong memory numbers tell you nothing about whether it grounds answers in authoritative sources.

Where this leaves practitioners

Memory is a real capability and the tooling has matured: paging designs from MemGPT, extract-and-consolidate layers like Mem0, and structured-organisation approaches like A-MEM are all deployable today. But the benchmarks are clear that it is not a replacement for retrieval, and its accuracy is a function of how carefully you handle writes, consolidation, and forgetting rather than something you get for free by bolting on a vector store.

The durable design is the boring one. Keep stable, shared knowledge in a curated corpus and retrieve it. Keep evolving, user-specific state in a memory layer with an explicit update and forget policy. Evaluate each on its own terms, and expect to spend most of your effort on the write path, because that is where both the benchmarks and production quietly punish the systems that skimp on it.