# Memory Tax paper: many agent memory often trails plain RAG > A new arXiv study tests 11 agent memory systems against naive retrieval on three benchmarks. Many lose, and the authors say why. URL: https://claudegpt.si/news/memory-tax-agent-memory-paper Published: 2026-10-28T22:52:49.328Z Author: ClaudeGPT Si Newsroom Section: Research Topics: Agent memory, RAG, Research, LongMemEval, Hugging Face Publisher: ClaudeGPT Si ## Key takeaways - Memory Tax (arXiv 2610.34073, October 25, 2026) tested 11 agent memory systems against naive RAG on three benchmarks. - The abstract says many dedicated memory systems underperform naive RAG in many cases, a negative net value. - The repository is public under MIT but its README still lists unfinished code and results. A new paper, ["Memory Tax: Dissecting the Net Value of Agent Memory"](https://arxiv.org/abs/2610.34073), finds that many dedicated agent memory systems do no better than plain retrieval over the raw chat history, and often do worse. The authors, led by Xuan He with Jiawei Han and ChengXiang Zhai among the co-authors, posted it on arXiv on October 25, 2026. It had 99 upvotes on [Hugging Face papers](https://huggingface.co/papers/2610.34073) as of October 28. ## What agent memory is A language model forgets everything between conversations. Agent memory is the layer that tries to fix that. It takes a long history of past chats or actions, stores something about it, and pulls the useful parts back into the prompt when the user asks a new question. The paper splits this into three steps: ingestion (turn the history into stored memory), retrieval (find the right pieces for a question) and generation (answer from them). Many products and open-source projects add clever structure at the first step. They summarize, extract facts, build graphs or sort items into tiers. The baseline is naive RAG, short for retrieval-augmented generation: skip the clever step, keep the raw history, and search it at question time. The paper asks what memory adds over that. ## Memory benefit and memory tax The authors call the gain from restructuring the history the memory benefit. They call the loss the memory tax: evidence that was dropped, blurred or made hard to find during ingestion. The net value is benefit minus tax, measured against naive RAG. Most of the paper is about the tax. They look at it three ways: - Faithfulness: was the information kept? - Accessibility: can retrieval find it? - Robustness: does the system hold up when the answering model changes? ## What they tested The team compared 11 memory systems: A-Mem, SimpleMem, MemOS, MemoryOS, LightMem, OpenViking, HippoRAG2, PlugMem, MAGMA, Mem-T and Memory-R2. Two retrieval baselines were run alongside them, with one and two rounds of retrieval, using hybrid keyword and embedding search with reranking. Each system was tuned to retrieve at least as much material as the one-round baseline, so none was handicapped by retrieving less. Three benchmarks were used: LongMemEval-s, PersonaMem-v2 and MINT-Eval, the last two as subsets. Answers came from three models: Qwen3.5-4B, Qwen3.6-27B and GPT-5.6 Luna at low reasoning effort, the last on a subset because of cost. A Qwen3.6-27B model judged the answers. ## What they found The abstract says that across multiple benchmarks and model backbones, many dedicated memory systems underperform naive RAG in many cases, which the authors read as a negative net value. They also say some systems beat the baseline in some settings, so the result is not uniform. The per-system score tables are in the paper, and we have not reproduced them here because we could not read the tables in full. The authors argue the losses come from the ingestion step, where a summary or extracted fact can drop or blur the evidence a later question needs. They also report that the memory systems' scores shift more than RAG's when the answering model changes, and that retrieval over the raw history often finds the evidence in fewer rounds. ## Beyond chat memory The paper extends the idea to two other kinds. Experiential memory stores what an agent learned from past tasks, tested on ALFWorld, WebShop, Mind2Web, BigCodeBench, AppWorld and tau2-Bench with Qwen3.5-4B. The authors say it can beat naive RAG on web tasks but often falls behind on tool use. World-model memory, used for consistent video or scene generation, did best when past observations were simply fed back in. For design, they suggest keeping access to the source history next to the memory, so a lossy summary can be corrected, and storing past queries and retrieval paths. ## Limits Some limits come from the authors, and some are ours to add. - The authors say their 11 systems are representative, not exhaustive, and that they followed each system's official configuration with limited tuning. They also changed a few settings, for example shortening SimpleMem's ingestion window from 40 to 10 utterances. - The benchmarks are static, offline and English-only. Long-running, multilingual use is untested. - Latent and parametric memory (stored inside model weights or caches) and multimodal memory are not covered. - Cost forced subsets of the benchmarks for some models. No stronger model was tested. - The authors note that simple retrievers already score high on LongMemEval-s, which leaves little room for memory to show a benefit. - The paper is a preprint, and we have not seen peer review. The result says these benchmarks and these systems show no net gain over good retrieval. It does not say memory is useless for products. ## Is the code out? Partly. The [MemoryTax repository](https://github.com/hexuan21/MemoryTax) is public under an MIT license. It holds the shared evaluation and judge code and a reproduction folder for each of the 11 systems. As of October 28, its README still lists unchecked items: results for the main experiments, ablations and reproduction code for each baseline. The quickstart also has a "TODO" for harness commands. Results are linked on [Hugging Face](https://huggingface.co/datasets/hexuan21/MemoryTax-Res), and there is a [project page](https://hexuan21.github.io/MemoryTax/). Expect the repository to change. If you build with agent memory, test it against plain retrieval over your raw history first. That is the comparison this paper makes. ## Frequently asked questions ### What is agent memory in AI? Agent memory is a layer that stores information from past conversations or tasks and brings the relevant parts back into a model's prompt later. Models do not remember earlier sessions on their own. ### What is the memory tax in the Memory Tax paper? It is the loss a memory system causes compared with searching the raw history directly: evidence that was dropped, obscured or hard to retrieve. The gain from restructuring is the memory benefit, and the paper finds the tax is often larger. ### Is agent memory better than RAG? Not on the paper's tests. Per the abstract, many memory systems underperform naive RAG in many cases, though some beat it in some settings. The results come from a preprint with subsets and English-only data. ## Sources - [Memory Tax: Dissecting the Net Value of Agent Memory](https://arxiv.org/abs/2610.34073), arXiv - [Memory Tax paper page](https://huggingface.co/papers/2610.34073), Hugging Face - [MemoryTax repository](https://github.com/hexuan21/MemoryTax), GitHub - [MemoryTax project page](https://hexuan21.github.io/MemoryTax/), Xuan He et al.