No Agent Passes the Three-Way Memory Test: Utility, Access Control, and Forgetting

Memory benchmarks for LLM agents have been built around a fiction: one user, one context, one trust level. Real deployments at hospitals, companies, schools, and households put multiple principals into the same memory pool, each with different roles, permissions, and deletion rights. No existing benchmark measures whether agents can handle that.

GateMem is built specifically for this gap. The benchmark jointly tests three properties that single-user evaluations treat as independent or ignore entirely: utility on long-horizon requests with state updates, access control across role-scoped authorization boundaries, and active forgetting after explicit deletion. These three properties are not separable in production. An agent that recalls accurately but leaks across role boundaries is not a memory agent. It is a liability.

The structure of GateMem reflects the messiness of real shared deployments. It spans medical, office, education, and household domains with long-form multi-party episodes, where memory is injected incrementally rather than loaded in a single context window. Hidden checkpoints verify whether deleted information has been purged. Structured judging and leak-target annotations make it possible to measure not just whether an answer is correct, but whether it reveals something it should not. The benchmark surfaces a specific failure mode that aggregate recall scores hide entirely: an agent can score well on utility while simultaneously leaking unauthorized or deleted content.

The results are direct. Across diverse baselines and backbone models, no method simultaneously achieves strong utility, reliable access control, and consistent forgetting. Long-context prompting produces the best governance scores, but at token costs that scale badly with episode length. Retrieval-based and external-memory methods cut that cost, yet both categories still leak unauthorized or deleted information at rates that disqualify them from institutional deployment. The gap between "works in a single-user benchmark" and "safe in a shared institutional setting" is not a matter of tuning. It is structural. For teams shipping agents into any multi-user environment, the takeaway is direct: your memory module has almost certainly never been tested for the failure modes that matter most in production.

We're thinking: Every enterprise AI deployment is, at its core, a multi-principal memory problem. A hospital assistant that remembers a patient's diagnosis should not surface that information to a billing clerk asking an adjacent question. An office assistant that stores a manager's performance notes should not echo them back to a direct report. We find it telling that the field has spent years refining retrieval quality while the authorization layer has gone essentially unmeasured. GateMem makes the problem legible, but the results also expose something harder to fix than an architecture choice: current agents have no reliable internal model of who is allowed to know what, and long-context prompting only papers over that gap at prohibitive cost.

Key takeaways:

• GateMem introduces a three-axis evaluation: utility on stateful long-horizon tasks, role-scoped access control, and verified active forgetting after deletion, tested jointly rather than in isolation.
• No baseline passes all three axes simultaneously; long-context prompting leads on governance but scales poorly with token cost, while retrieval and external-memory methods remain cheaper but leak unauthorized or deleted content, with results consistent across multiple backbone models and four deployment domains.
• Teams deploying shared assistants in any multi-user setting, including internal enterprise tools, should run their memory module against GateMem's access control and forgetting checkpoints before any production rollout, not just against recall or utility metrics.

Source: GateMem: Benchmarking Memory Governance in Multi-Principal Shared-Memory Agents