Measuring Agent Memory — The Backdrop and Scope of the Evaluation Benchmarks

TL;DR

• Within a single week, three agent-memory evaluation benchmarks landed in the same place at once. LongMemEval-V2 (2026-05-13, UCLA) · MEME (2026-05-13, KAIST + Tübingen + NAVER) · MemPrivacy (2026-05-10, MemTensor + Tongji). Same category, but each measures an orthogonal axis — accumulation (LongMemEval-V2), dependency reasoning (MEME), and the privacy-utility trade-off (MemPrivacy).
• All three stake out territory existing benchmarks didn't measure. LongMemEval-V2 measures environmental experience accumulation (can a web agent become an "experienced colleague"), MEME measures multi-entity dependency reasoning (does a change to one entity propagate to the entities that depend on it), and MemPrivacy measures utility loss by privacy tier. In the same moment, three different blank spots were filled at once.
• Key finding — all three found where the systems break. LongMemEval-V2: even the best memory system (AgentRunbook-C) tops out at 48.3% on Environment Gotchas. MEME: every system collapses simultaneously on Cascade (dependency propagation) 3% · Absence (uncertainty recognition) 1%. Claude Opus 4.7 can barely solve it partially — at 70× the cost. MemPrivacy: a specialized model arrives that handles in 0.4–2 seconds what general LLMs take 2–96 seconds to do.
• The appearance of measurement tooling is the signal that the category has solidified. What defines this category's current surface area is not the intersection but the large union of the three benchmarks. The mere fact that three landed at once is the first objective evidence that academia has recognized memory as its own domain — the same threshold the Agent Framework category once crossed.

1. Why the Arrival of Measurement Tooling Is a Signal

Pulling back the definition laid out in the category index — agent memory is the answer to what to accumulate, and how, outside the context window. Across 2025 through Q1 2026, every metaframe arrived carrying its own answer, but for those answers to be comparable to each other, you need a common measurement tool. Until measurement tooling existed, only comparison of claims was possible — comparison of evidence was not.

That the three benchmarks landed in the same single week is the crux. It's the first objective evidence that academia has begun recognizing the category as its own domain, and the simultaneous arrival is itself a convergence signal. Not spontaneous coincidence, but a sign that the category has reached a measurable stage.

The three benchmarks fill different blank spots. Where LongMemEval V1 (2025) stopped at conversational memory (user-assistant chat), LongMemEval-V2 extends to environmental memory (web/work agents); MEME catches the dependency reasoning that RULER, LoCoMo, and MemBench couldn't; and MemPrivacy catches the privacy-utility trade-off that none of the existing benchmarks touched. That the three are orthogonal raises the value of the union.

2. LongMemEval-V2 — "Can It Become an Experienced Colleague"

2.1. What It Measures

The question UCLA (Di Wu et al., 2026-05-13) catches with LongMemEval-V2 is simple — can a memory system turn an agent into a "colleague with environmental experience"? Where the original V1 (2025) stopped at conversational memory (user-assistant chat), V2 extends to experience accumulation in web-environment agents.

The body of the benchmark is 451 hand-curated questions + a 100–500 trajectory haystack + up to 115M tokens. It directly measures five core memory capabilities:

The five axes correspond to different kinds of forgetting. If the same user does well on Static State Recall but collapses on Premise Awareness, you can pinpoint exactly which part of the memory system is empty.

2.2. How It Measures — Context Gathering Formulation

The design crux of V2 is the directness of measurement. Where many existing benchmarks measured memory indirectly through task success, V2 does:

Insert(trajectory) → Query(question) → Truncated_Context(200K) → Reader_LLM → Answer

A fixed reader LLM (Qwen3.5-9B, 200K context truncation) answers using only the context the memory system returns. The reader is fixed so that the quality of the memory system is reflected directly in reader performance. Identical to the interface a downstream agent would actually use.

The scale is overwhelming too — 25M (Small) to 115M (Medium) tokens. That's 15–100× past V1's 1.5M max. To ensure the answers can't come from parametric knowledge, the data is pre-filtered with frontier LLMs (Gemini-3-Pro, GPT-5.2, Grok-4.1, Claude-Opus-4.6).

2.3. Key Finding — Where the Coding Agent Becomes the Memory Controller

The V2 body also proposes two baseline memory systems:

• AgentRunbook-R (RAG-based) — three knowledge pools (raw states, events, notes), Qwen3-Embedding-8B + a Qwen3.5-9B controller. 58.6% (Small).
• AgentRunbook-C (coding agent-based) — keeps every trajectory as disk files and has a Codex (GPT-5.4-mini, xhigh reasoning) sandbox gather evidence directly. 74.9% (Small).

The most important result is that the coding agent catches what RAG drops. RAG is a single query → vector similarity model, so it can't pull workflow and procedure across multiple layers. The coding agent explores the file system directly, so it can refine the evidence.

But the per-capability breakdown shows Environment Gotchas is the weakest at 48.3% — meaning even the best system manages only half on "recognizing environment-specific failure patterns." The signal: memory is strong at remembering what it knows but weak at recognizing what it ought to know.

The cost is laid bare honestly too — AgentRunbook-C runs 108 seconds per query. That's 4× slower than AgentRunbook-R (27 seconds) but 17 points more accurate. Drawing a new frontier in the latency-accuracy trade-off is V2's direct contribution.

2.4. V1 → V2 — A Beat in the Category's Evolution

Compress the V1-to-V2 difference into one line: from conversational memory to environmental memory. V1 measured "what did this user say last time," V2 measures "does it remember how to operate in the environment this user works in." The unit of measurement shifted from utterance to trajectory. It's the moment the academic measurement tool arrived at the same place as the metaframe designs (GSD .planning/, gstack /learn, ECC instinct) that all treat environmental memory as a first-class citizen.

3. MEME — "Where Dependency Reasoning Breaks"

3.1. What It Measures

MEME from KAIST + Tübingen + NAVER (Seokwon Jung et al., 2026-05-13) — Multi-entity & Evolving Memory Evaluation. The question it catches is narrower and sharper — does a change to one entity propagate correctly to the entities that depend on it.

Existing memory benchmarks (RULER, LoCoMo, LongMemEval, MemBench) measure only single-entity updates. But real agents handle dependent knowledge — when a user moves to a new city, the commute time, nearby facilities, and home prices all change. One change produces many consequences.

MEME defines six tasks along two orthogonal axes:

Of these, three tasks — Cascade, Absence, Deletion — are territory no existing benchmark measures.

• Cascade: propagate a change per dependency rules — if the parent entity changes, the child must too.
• Absence: when no resolution rule exists, recognize the uncertainty — can it report "the old value is invalid but the new value is unknown"?
• Deletion: verify that an explicitly deleted fact is no longer reported.

3.2. The Honesty of the Dataset

MEME's dataset is hand-built as a DAG-based knowledge graph. Two domains (Personal Life: 39 entities, 34 edges, 5 patterns; Software Project: 51 entities, 27 edges, 6 patterns). 100 episodes (~35K tokens/episode), 694 evaluation questions per episode. Verifiable ground truth by construction — the answer is unambiguous by design.

A GPT-4o judge shows 98.6% agreement with human annotators (meta-validation). In other words, not a subjective-evaluation game but a benchmark with objectively specified answers.

3.3. Key Finding — Every Memory System Collapses Simultaneously on Dependency Reasoning

The six systems MEME measured:

• Raw retrieval: BM25 · text-embedding-3-small
• LLM-processed: Mem0 · Graphiti (temporal knowledge graph)
• File-based agents: MD-flat (markdown + tool-use) · Karpathy Wiki (daily log → compiled topic)

Results:

Cascade averages 3%, Absence 1%. Every system collapses at once. Attempts at partial mitigation — raising retrieval depth from k=5 to 40, prompt optimization with DSPy SIMBA, upgrading the answer LLM to Sonnet 4, dialing filler noise to 0/32K/128K — none had any effect.

3.4. The Three Stages of Failure — encoding, retrieval, answering

The failure-mode analysis MEME laid out honestly:

1. Encoding ✓ — dependency rules, pre-change values, and change events are all stored well.
2. Retrieval ✗ — the change event loses in ranking to the pre-change value. The vector retriever preferentially returns the semantically closer pre-change value. Tool-use/graph also fails to surface the change event.
3. Answering ✗ — even when k=20/40 retrieves both rule and change, the LLM fails to propagate — it reports the stale value as-is.

So all three stages have a problem. The diagnosis: dependency reasoning isn't solved by fixing retrieval alone.

3.5. The Only Working Solution — Self-Propagation at 70× the Cost

The only partially working mechanism MEME found: MD-flat + Claude Opus 4.7. Reaches Cascade 0.32, Absence 0.59.

The mechanism is simple — Opus explicitly writes a "Contingency:" entry at ingest time and, on an upstream change, writes the propagated value directly into memory. That is, self-transformation into a form the retriever can find.

The cost — $4.54 per episode vs. baseline gpt-4.1-mini $0.065 = 70×. And the side effect — Exact Recall drops from 0.94 to 0.60 and Tracking from 0.77 to 0.20. The trade-off: push active writes up to catch dependency reasoning and basic retrieval gets muddied.

In one line: as of 2026-05, no memory system solves dependency reasoning at a practical cost. MEME is the first benchmark to lay this spot bare honestly.

3.6. Ripple Effect — The Link to the Knowledge-Editing Literature

The ripple effect MEME catches is the place where the core challenge of the knowledge-editing literature gets carried into memory systems. Knowledge editing — which directly modifies LLM weights — deals with how editing one fact propagates to dependent facts, and MEME measures that the same challenge arises in external memory systems too. The spot where two fields tackle the same problem but have evolved separately.

4. MemPrivacy — "The Trade-off Between Privacy and Utility"

4.1. What It Measures

MemPrivacy from MemTensor Shanghai + Tongji University (Yining Chen et al., 2026-05-10). Where LongMemEval-V2 and MEME measure the memory system itself, MemPrivacy measures the privacy surface of edge-cloud agents.

The problem setup — on edge devices (personal phones, laptops), sensitive information is sent to a cloud LLM, and prompt injection, multi-turn memory attacks, and leakage attacks succeed at a 69–75% rate. The existing countermeasure is raw masking (substituting ***), but its utility loss is 26–41% — protect privacy and the task breaks.

What MemPrivacy measures — the Pareto frontier of privacy leakage risk vs. utility loss:

min ℛ_priv(ℱ_san) subject to ℒ_util(ℱ_san, ℱ_res) ≤ ε

4.2. A Four-Tier Privacy Classification

Sensitivity classification as PL1 ~ PL4:

The user can choose how far up the tiers to protect. The base design assumptions — protect PL4 only and utility loss is <0.18%; protect all of PL2-PL4 and it's within 1.6%.

4.3. The Mechanism — Reversible Pseudonymization via Typed Placeholders

MemPrivacy's crux is not masking but typed placeholders:

• Raw Masking: "the password is secret123" → "the password is ***" (meaning destroyed)
• Untyped Placeholder: "the password is secret123" → "the password is <MASK_1>" (type lost)
• MemPrivacy: "the password is secret123" → "the password is <PASSWORD_1>" (type preserved, value protected)

Because the type information is preserved, the cloud LLM can generate a natural response, and the original is stored in the edge device's local DB to restore the placeholder back to the original in the response. That is, an architecture-level isolation where even if cloud-side leakage occurs, only the placeholder is exposed.

4.4. Key Finding — A Specialized Small Model Overwhelms the Frontier LLM

MemPrivacy 4B-RL scores F1 85.97% on MemPrivacy-Bench, +7.56 points over the best general LLM (Gemini-3.1-Pro) at 78.41%. The more striking spot is latency:

50–200× faster. Even the 0.6B model is more accurate than most general LLMs — a spot where task-specific specialization overwhelms the generalist.

This is because MemPrivacy was trained with SFT + GRPO RL — GRPO directly optimizes the precision-recall balance as a reward. An ablation showing SFT alone can't get there is provided too.

4.5. An Honest Accounting of Utility Loss

MemPrivacy was measured layered atop three memory systems — LangMem, Mem0, and MemoBase:

Raw masking renders the memory system effectively useless (-26 to -42%). MemPrivacy comes in within -1.6% at an equivalent protection level. A place where the privacy-utility trade-off is honestly resolved.

4.6. Position in the Category — Where It Meets Governance

You should register that MemPrivacy is not merely a privacy tool but a place where it meets the governance category. The audit, approval, and compliance surface laid out by the Agent Governance category is the place that reappears in persistent memory. When memory in operator domains (ECC's healthcare/logistics/finance skill clusters) accumulates PII, PHI, and financial information, it shows through measurement that a mediation layer like MemPrivacy is needed.

5. Where the Three Don't Measure — The Category's Blank Spaces

§1 said that the three benchmarks being orthogonal raises the value of the union. So where, then, is the spot none of the three covers? The category's current blank surface area shows through here.

5.1. Multi-user shared memory — none

All three benchmarks are single-user scenarios. But for cross-user sharing mechanisms like gstack's domain-skill promote-to-global or ECC's /instinct-export and /instinct-import, the core question is whether each one's trust weighting is still valid in a different environment. No measurement tool exists.

5.2. Edit/revert workflows — none

The correctness of workflows where a user directly edits, deletes, or reverts memory. The Anthropic Memory tool allows file-level editing, but post-edit consistency isn't measured. MEME covers a Deletion task, but it only measures deletion the system handles automatically — user-driven editing is a blank spot.

5.3. Temporal-decay accuracy — only partial

Time-based memory weakening mechanisms like ECC instinct's confidence decay or GSD .planning/'s milestone closure — which information should weaken when. MEME handles explicit change via Tracking and Deletion, but information that should weaken by time alone (e.g., "the library this user liked last year") is a blank spot.

5.4. Team-level consistency — none

In team adoption scenarios like gstack team-mode or the ECC GitHub App (150+ installs), memory sync and conflict resolution across team members. Whose instinct is canonical, and the priority of team wiki vs. personal memory. MemPrivacy handles personal protection, but the balance of team sharing vs. personal protection is a blank spot.

5.5. Vertical domain-specific evaluation — none

In operator workflows (ECC vertical skills §3.2), domain-specific memory evaluation for healthcare HIPAA, finance PII, customs trade compliance. LongMemEval-V2 uses the general WebArena/WorkArena environments, MEME uses Personal Life/Software Project, MemPrivacy uses general PII — regulated-domain vertical evaluation is a blank spot.

These five spots are where the next six months' follow-on memory measurement tooling seems likely to appear. If the simultaneous arrival of benchmarks announces the formation of the category, the blank spots between benchmarks announce the next entry.

6. Synthesizing the Three — A Division-of-Measurement Table

Organizing the three benchmarks by the division of what they measure:

The common implications of the three measurements:

1. Different memory systems are strong in different places. The coding agent is strong on LongMemEval, but on MEME MD-flat is strongest. That is, the best system differs by capability axis — no single-best memory system exists.
2. Specialized memory models that don't depend on frontier LLMs are emerging. MemPrivacy 4B-RL is the first instance — a small model trained task-specifically overwhelms the generalist. A signal that the category has reached a stage where training a model that's good only at memory tasks is meaningful.
3. The cost-accuracy trade-off is honestly exposed. Coding-agent memory control (LongMemEval-V2 AgentRunbook-C: 6.9× cost), Opus 4.7 active propagation (MEME: 70× cost), the specialized model (MemPrivacy: 50–200× faster). Each mechanism has a different cost surface.

7. Synthesis — The Mere Appearance of Measurement Tooling Is the Category Signal

In three lines:

1. That three benchmarks landed in the same single week is the signal of category formation. LongMemEval-V2 (5/13) · MEME (5/13) · MemPrivacy (5/10) — not spontaneous simultaneity but the first objective evidence that academia has begun recognizing memory as its own domain. Just as the Agent Framework category formed from the simultaneous arrival of four metaframes, the Agent Memory category reached the measurable stage with the simultaneous arrival of three benchmarks.
2. The three fill different blank spots — orthogonal measurement. LongMemEval-V2 covers environmental experience accumulation, MEME dependency reasoning, MemPrivacy the privacy-utility balance. The three have a small intersection but a union that defines the category's current surface area. On which benchmark you should measure the metaframes' memory mechanisms (Superpowers · GSD · gstack · ECC) is now separated out.
3. All three found where things break. LongMemEval-V2's Environment Gotchas at 48.3%, MEME's Cascade at 3% and Absence at 1%, MemPrivacy's raw masking at 26–42% utility loss. The honesty of the benchmarks surfaces the limits of the memory systems and points to where the next mechanism goes.

Operating conclusion: when adopting a metaframe, be conscious that which aspect should be evaluated is now separated out by benchmark. If your usage is heavy on environmental tasks, prioritize a system that does well on LongMemEval-V2 (coding agent + scaffolding). If your domain needs multi-entity dependency reasoning (e.g., project management, relational data), be conscious of the 70× cost trap MEME found. If you're in a vertical handling PII/PHI, a typed-placeholder mediation layer like MemPrivacy must replace raw masking. A benchmark is a tool to verify marketing claims, not the body that defines the category — but the fact that the three landed in the same single week makes clear that the category has entered a stage of objective measurability.

One-line recommendation — if your memory system has to be evaluated, first ask which of the three your work's axis is closest to. Environmental experience accumulation (LongMemEval-V2), dependency reasoning (MEME), and privacy balance (MemPrivacy) each measure a different kind of forgetting. The odds that the same memory system is strong on all three are slim. Choosing the benchmark is itself the first decision of memory evaluation — and this piece lays out the three coordinates of that decision.

References

Primary sources

• LongMemEval-V2 paper — UCLA, Di Wu et al., 2026-05-13
• LongMemEval-V2 project site — dataset · code · leaderboard
• MEME paper — KAIST + Tübingen + NAVER, Seokwon Jung et al., 2026-05-13
• MEME project site — code · data
• MemPrivacy paper — MemTensor + Tongji, Yining Chen et al., 2026-05-10
• MemPrivacy GitHub — code · models · MemPrivacy-Bench
• MemPrivacy models on HuggingFace

Category index + series

• Agent Memory category — the 4-axis (Capture · Storage · Retrieval · Lifecycle) decomposition
• Agent Framework series — the pieces where the four frames' memory mechanisms are embedded
• Agent Governance category — the governance/audit surface of persistent memory

Prior benchmarks (the spot V2 extended)

• LongMemEval V1 (NAACL 2025) — Di Wu et al., conversational memory measurement
• RULER · LoCoMo · MemBench — the comparison targets in MEME's Table 1