Why Large Language Models Need Memory: Understanding the Three-Dimensional Framework of Memory for Large Language Models

Large language models are moving beyond generating answers from the current context alone. Increasingly, they are expected to accumulate experience across tasks and conversations.

But where should information live? When should it be updated? And how long should it persist? The survey paper Memory for Large Language Models provides a clear architectural answer to these questions.

The paper was written by researchers from Tsinghua University, NUS, and Bosch AI. Its central argument is simple:

Memory for large language models is not merely a longer context window. It is an architectural capability that must be explicitly designed, updated, and managed.

This article introduces the paper’s main ideas and reasoning process. I also conducted a self-test experiment based on its methodology, which validates the effectiveness of the three-dimensional memory framework described in the paper. Interested readers can read it here.

1. Why is a context window not the same thing as memory?

Traditional Transformers use attention to read prior text and retain the Keys and Values of historical tokens in a KV cache. This can be understood as a form of short-term working memory.

However, it has clear limitations. The longer the context, the higher the computational and memory cost. Information usually does not persist naturally after a conversation ends. And the model has limited ability to decide what is worth retaining in the long run.

flowchart LR
 A[Input history] --> B[Attention / KV Cache]
 B --> C[Current answer]
 B -. Usually expires when the context ends .-> D[Short-term memory]
 E[Long-term tasks and experience] --> F[Writable, retrievable memory module]
 F --> C
 F --> G[Cross-session retention]

The paper therefore seeks to turn memory from a by-product of context processing into an independent design variable in a model.

2. A three-dimensional framework for understanding LLM memory

The paper argues that every memory mechanism can be understood through three independent dimensions.

1. Representation: implicit and explicit

Implicit memory is tightly coupled with a model’s forward computation and usually has no separate read/write interface. Typical examples include KV caches, RNN hidden states, and the recurrent states used by models such as Mamba and RWKV.

Explicit memory, in contrast, is a separate storage component that a model can deliberately read, write, and update. Examples include vector indexes, key-value stores, memory slots, and writable parameter modules.

• Implicit memory is like the state naturally retained inside a model’s mind.
• Explicit memory is like a notebook the model maintains and can query on demand.

2. Update mechanisms: offline and online

Offline updates happen during pretraining or fine-tuning. Once training is complete, the model’s knowledge is largely fixed: stable, but not especially flexible.

Online updates allow a model to modify its state, parameters, or external memory during inference and deployment. A recurrent state, for example, is continually updated as new input arrives; some test-time training methods let a model make local adaptations from new data.

Online updates improve adaptability, but they can also introduce error accumulation, overwriting of older knowledge, and memory drift.

3. Persistence: short-term and long-term

Short-term memory serves the current context or a single inference run. The KV cache is a classic example.

Long-term memory persists across tasks and conversations. It may live in writable parameters, a separate retrieval database, or a dedicated memory module.

flowchart TB
 A[Short-term memory: current question and recent context] --> B[Filtering and compression]
 B --> C{Worth retaining long term?}
 C -- No --> D[Expire or discard]
 C -- Yes --> E[Long-term memory: facts, rules, and experience]
 E --> F[Retrieve on demand for later tasks]

3. What technical approaches does the paper cover?

Rather than grouping systems only by model name, the paper places different techniques inside this common memory framework.

1. Attention and its extensions: KV cache, sliding windows, sparse attention, and selective attention, all focused on reducing the cost of long contexts.
2. Recurrent and state-space models: Mamba, RWKV, and related approaches that compress history into a continually updated state instead of preserving every token.
3. Writable parametric memory: specialized modules that write information during training or inference to support continuous adaptation.
4. Retrieval-based memory: key-value tables, vector indexes, or dedicated data stores that retrieve relevant information when needed.
5. MoE: selecting different expert parameters according to the input, which can be viewed as a form of conditional memory.
6. Hybrid architectures: combining attention, recurrent state, retrieval, and expert networks so that different mechanisms serve different time scales.

These techniques address the same underlying challenge: how to store, update, and invoke historical information at low cost while avoiding forgetting and interference.

1. The three dimensions are not mutually exclusive

The value of the framework is that it prevents us from labeling a technique too simply as either “having memory” or “not having memory.” A single system may use several kinds of memory at once. The KV cache in a chat window is implicit, online, and short-term; a database of user preferences is explicit, online, and long-term; general knowledge encoded in pretrained parameters is closer to implicit, offline, and long-term memory.

The design question is therefore not whether to give a model memory. It is which representation, update frequency, and retention period best fit a particular kind of information.

For example, a filtering condition a user has just mentioned must immediately support the current answer, so it belongs in working memory. A business metric definition that will be reused repeatedly belongs in a retrievable long-term knowledge base. Broad knowledge such as language ability and industry common sense is better supplemented through training or retrieval than repeatedly written into every conversation.

2. What does each route solve best?

From an engineering perspective, these mechanisms do not have equal priorities.

Real-world systems therefore rarely rely on one route alone. A customer-service agent can use a context window for the current conversation, a retrieval store for product rules, and a user profile for long-term preferences. A data analysis agent should place greater emphasis on verifiable explicit memory, because it works with changing schemas, metric definitions, and business decisions.

There is also a practical rule of thumb: if information must be traceable, auditable, correctable, or maintained by multiple people, it should not live only in a model’s implicit state. It should be stored explicitly with provenance and versioning. The model can understand and organize information, but it must always be possible to answer where a fact came from.

4. Implications for memory management in OpenClaw Data Agents

The paper is highly relevant to Data Agents built on OpenClaw.

The most important principle is:

Memory should preserve how to understand and use data; the data warehouse should preserve the data itself.

Raw data, temporary SQL results, and long-term business definitions should not all be placed in one memory file. A better approach is to separate them by persistence and trustworthiness.

flowchart TB
 U[User question] --> W[Working memory]
 W --> Q[Data warehouse / database query]
 Q --> R[Temporary results and analysis process]
 R --> V{Verified and reusable?}
 V -- No --> X[Expire when the session ends]
 V -- Yes --> K[Long-term knowledge layer]
 K --> M[Metric definitions, schemas, quality rules, business decisions]
 K --> P[Source, owner, update time, and scope]
 M --> N[Retrieve and cite in later tasks]

A Data Agent’s memory can be separated into four layers:

For OpenClaw, existing capabilities can be mapped to these layers:

• MEMORY.md: stable, concise, confirmed business rules and long-term decisions only;
• daily memory files: analysis processes, temporary observations, and leads that still need validation;
• memory_search: retrieve relevant context when needed instead of inserting all history into every prompt;
• databases or warehouses: always remain the single source of truth for raw data;
• knowledge wikis or data catalogs: store schemas, metric definitions, and data-quality rules together with sources and versions.

From recording information to usable memory

The easiest mistake in a memory system is to confuse saving information with creating capability. A note only becomes usable memory when it can be accurately retrieved, correctly interpreted, and safely updated later.

For a Data Agent, each piece of information can move through five steps:

1. Capture: identify candidate information from user confirmations, data dictionaries, task conclusions, or human feedback.
2. Validate: confirm the source, definition, applicable product scope, and time range; distinguish facts from assumptions and observations.
3. Classify: decide whether it belongs to temporary context, a project decision, a business rule, or an experience pattern.
4. Write: store the content together with its source, owner, update time, and conditions for expiry.
5. Retrieve and review: cite it when relevant in later tasks, then update, replace, or retire it as data and rules change.

Validation and classification are usually more important than the act of writing. “D0 collection entry rose for a channel this week,” for instance, is an observation and should not become a long-term rule. The standard formula for D0 collection entry and an approved channel-mapping rule, by contrast, can become long-term knowledge. Mixing the two causes an agent to treat one-off fluctuations as durable patterns.

Give memory a sense of time

Information in data analysis is rarely permanent. Channel names may change, metric definitions may be revised, a table may be migrated, and an old conclusion may be overturned by new data. Long-term memory should therefore include at least four kinds of metadata:

• Source: data dictionary, business owner, meeting record, or automated analysis;
• Effective date: when the rule began to apply;
• Last updated: when it was last confirmed;
• Expiry condition: which version change, product change, or data reconstruction requires a review.

With this information, an agent can do more than give the formula when asked how a metric is calculated. It can also explain the product scope and version for which the rule applies. When a question concerns old data, it can proactively flag that the definition may differ. This sense of time is what separates long-term memory from a simple notebook.

It is especially useful to establish a threshold for writing to long-term memory:

1. Does the conclusion come from a trustworthy source?
2. Has it been validated or confirmed by a human?
3. Are its source, update time, and scope recorded?
4. Does it have an expiry condition or a newer version?

Only information that passes these checks should be promoted from working memory to long-term memory. This prevents a Data Agent from treating an accidental query result, an obsolete definition, or a model guess as a fact for future work.

A lightweight but actionable implementation

There is no need to begin with a complex vector database or writable parameter module. For most teams, a three-layer structure of short-term files, a long-term rule base, and the original data source already covers many needs. Keep conversations and daily tasks in short-term files; write confirmed rules into long-term documents or structured tables; and always return to the database for numerical facts.

As the volume of content grows, add retrieval to the long-term rules and create dedicated knowledge pages for recurring questions. This lowers the initial build cost and avoids over-automating before the rules are stable. A mature memory system is not built in one pass. It emerges gradually through repeated validation, use, and retirement.

Conclusion

The paper’s greatest value is not that it proposes one new model. It provides a common language.

KV Cache, Mamba, RAG, writable parameters, MoE, and test-time training may look like separate directions, but they all answer the same questions: what should a model remember, how should it update that memory, how long should it keep it, and when should it forget?

For agent systems, memory management should not mean “store as much as possible.” It should mean storing and invoking information at the right layer, with the right update rule, for the right duration.

In other words, a good memory system must both remember well and forget well. It should retain information that is stable, verifiable, and useful for the next task; allow temporary results to expire naturally; replace stale rules; and keep unconfirmed inferences out of the fact base. For LLMs and agents, this bounded form of memory is often much closer to reliable intelligence than an endlessly expanding context window.

Original paper: Memory for Large Language Models