Agent Memory Lab

Context

Agent memory shipped hard in 2026. ChatGPT now rewrites its memories of you in the background, Google's Gemini Spark runs as an always-on personal agent, and Mem0, Zep and LangMem sell memory as infrastructure. Vendors publish big improvement numbers.

At the same time, people who use these tools every day report the same thing: memory turns into noise. PAST-Bench (August 2026) sharpened the point. A later-task gain can be real and still not be caused by the memory the framework takes credit for.

Why this is a product question

What an assistant writes down, when it writes, what it throws away and what replaces what: that is a spec. Somebody decides it, engineers implement it, users live with it. This project makes that spec measurable.

The first question

When an assistant reflects on a session, does it write a rule it can use next time, or a description of what happened? "G shortened my reply to Priya" is true and useless tomorrow. "Keep replies short" helps with every email after it.

My hypothesis: mostly descriptions, because the reflection prompt asks what went wrong, and describing is the path of least resistance. If that holds, every gain built on top of memory is resting on sand.

What I found

I was half wrong. Asked what went wrong, the assistant wrote descriptions 41% of the time and instructions 53% of the time, not the landslide of diary entries I expected. Change one line of the reflection prompt to ask for rules instead, and instructions go to 85%. The prompt is the lever, and it is one sentence long.

The catch is that shape and usefulness are different things. Handed to a fresh assistant on emails it had never seen, instructions changed behaviour 46% of the time from the first prompt and 45% from the second. Asking for rules produced far more rule-shaped notes without making them any better. Descriptions did worse, 28%, but not nothing, and all of it sits well above the 10% of irrelevant control notes ("G uses a MacBook") that also nudged the model.

Across both prompts, 105 of 251 notes changed behaviour. The other 146 were written, stored, reread every session, and did nothing.

One number I did not go looking for: the memory that curated itself hardest ended up almost empty. The rule-framed assistant wrote 123 notes over 20 sessions and kept 2. The other kept 20. Nobody sees that happen inside a shipped product.

How you classify a note decides the story. An earlier version of the labeler read plain instructions as descriptions and put the first figure at 66%. The numbers here come from the labeler that agreed with my own blind labels on 8 of 10 notes (Cohen's kappa 0.64), and that agreement rests on 10 items, which is thin.

What I would ship

Ask for rules, not a post-mortem. One sentence in the reflection prompt moved rule-shaped notes from 53% to 85%. Cheapest change in the system, and it costs nothing at inference time.

Make a note earn its place. Before a note is written to memory, run it against a handful of held-out cases and keep it only if it changes the output. On this run that gate would have discarded 58% of the store, which is 58% less noise to reread forever.

Publish the floor with the number. Any note in the prompt nudges the model a little. Without a control, 10% of a memory system's apparent effect is just having text in the context window. Every improvement claim should carry its floor.

Watch the curator, not just the writer. Extraction gets the attention; the delete step is where memory quietly disappears. Log every add, update and delete, and alert when a store shrinks faster than it grows.

How the test works

• An email assistant drafts replies for one scripted user, G, across 20 sessions. G has five hidden style rules: two stated out loud ("Too long."), three only shown through G's own edits, the way you quietly fix a draft before sending it.
• After each session the assistant reflects, then curates its memory with add, update or delete, the two steps Mem0 published.
• Two reflection prompts, 20 sessions each, same emails, same seeds. One asks what went wrong, one asks for rules.
• Every note gets two verdicts: a type (episodic, semantic or procedural, labelled by a different model and checked against my blind labels) and a transfer test, where the note alone is handed to a fresh assistant on five unseen emails.
• A pilot ran first with no memory at all, to make sure the rules were not already being followed. Three of the original five were, so they were replaced.

Decisions and tradeoffs

A chat assistant, not a coding agent. Chat is where most people meet memory, and where the noise complaints come from. Writing is also one of the most common things people ask AI to do, and still one of the most disappointing.

A scripted user. A second model playing the user doubles the noise and the runtime. The cost is conversations that are less natural than real ones.

The pilot rewrote the rules. Before the real run, I checked how often the assistant followed each rule with no memory at all. Three of the original five passed 95 to 100% of the time. A rule it already follows cannot show learning, so I replaced them with rules it breaks by default.

A local 8B model. Zero API spend, and the point is the difference between conditions, not raw capability. A larger model may extract differently, and I say so.

What's next

• Attribution: memory on, memory off, and memory scrambled (same length, useless content), to separate the memory from the extra context.
• Decay: change a rule halfway through and count how many sessions the dead lesson survives.
• A dinner concierge where preferences are only revealed through rejections, closer to what Gemini Spark does with OpenTable.