When Agent Memory Becomes Noise: Evaluating Stale and Conflicting Context

AI agents are becoming better at working across multiple steps, sessions, and tasks. But as agents take on longer-running work, another problem becomes increasingly important:

What happens when the memory they retrieve is no longer correct?

A memory system may successfully retrieve information from the past. However, that does not necessarily mean the information is useful for the current task.

An old design decision may have been replaced. A debugging approach may have already failed. A user preference may no longer apply. Two historical records may provide conflicting evidence.

In these cases, retrieving more context can make an agent less reliable rather than more capable.

Retrieval is only the first step

Many memory systems are evaluated by whether they can retrieve relevant information. This is an important capability, but it does not fully describe how memory affects an agent.

A retrieved memory also needs to be:

• relevant to the current task;
• consistent with newer evidence;
• valid in the current project state;
• understandable to the downstream agent;
• useful for completing the task.

Consider a coding agent working on a long-running repository. It may retrieve a previous implementation decision that was correct several months ago. Since then, the architecture may have changed. If the agent treats the old decision as current, the memory system has introduced a failure mode.

The problem is not that retrieval failed. The problem is that the system did not understand the status of the retrieved information.

Three difficult types of memory

Stale memory

A stale memory was once correct but is no longer valid.

Examples include:

• an old API specification;
• a deprecated configuration;
• a previous product requirement;
• a bug that has already been fixed;
• a design decision replaced by a newer approach.

A useful memory system should be able to recognize that time and state matter.

Conflicting memory

Different historical records may disagree.

For example, one project document may say that a service uses one database, while a later engineering decision moves it to another. Both records may be retrieved for the same query.

The agent then needs to determine which evidence is newer, more authoritative, or more relevant to the current task.

Noisy history

Long-running projects generate a large amount of information that is not relevant to every task.

A repository may contain unrelated commits, abandoned experiments, failed approaches, and duplicated discussions. If the memory system returns too much of this history, the agent may spend more effort interpreting context than solving the problem.

This is particularly important for Coding Memory. The goal is not to retrieve the entire history of a repository. The goal is to retrieve the right engineering experience for the current task.

What should a benchmark measure?

A more complete Agent Memory evaluation should test not only retrieval, but also downstream task performance.

Useful questions include:

1. Can the system retrieve information related to the task?
2. Can it distinguish current information from outdated information?
3. Can it handle conflicting evidence?
4. Can it remain useful when irrelevant history is added?
5. Does the retrieved memory improve the final answer or implementation?
6. Does it reduce repeated explanations and failed attempts?

These questions require controlled evaluation conditions. Otherwise, it becomes difficult to determine whether an improvement comes from the memory system, the language model, the prompt, or the evaluation process itself.

A shared evaluation framework

Agent Memory Leaderboard (AML) provides a common evaluation process for different memory systems.

Participants provide their own memory infrastructure through standardized Add and Search interfaces. The AML platform then handles downstream Answer, Evaluation, review, and public reporting.

This setup allows different architectures and retrieval methods to be compared within a shared evaluation boundary.

The second AML cycle covers three areas:

• Textual Memory for long conversations and cross-session history;
• Coding Memory for software engineering tasks and repository experience;
• Multimodal Memory for systems working with text and image-based evidence.

The purpose is not to define one correct memory architecture. It is to make the strengths and weaknesses of different approaches easier to observe and discuss.

From remembering more to remembering better

A reliable agent does not need to remember everything.

It needs to:

• retain information that remains useful;
• identify when an old memory has changed;
• resolve conflicts between historical records;
• ignore irrelevant context;
• apply past experience to new tasks.

This is why Agent Memory evaluation should move beyond simple retrieval accuracy.

The more important question is whether memory helps an agent make better decisions under changing conditions.

AML Cycle 2 is currently open to researchers, developers, open-source teams, commercial AI teams, and individual contributors. The evaluation covers Textual, Coding, and Multimodal Memory.

Complete submissions are due by October 31, 2026, at 23:59 UTC+8.

Learn more and participate:

https://agentmemoryleaderboard.ai/competition