The Missing Science of Semantic Memoization

Nothing about a CPU cache looks like it should work. You put a memory thousands of times smaller than the address space in front of the real one and claim most requests will be served from it. The claim happens to be true — but only because programs have a property nobody designed into them: they keep touching what they touched recently. Locality was not derived from first principles; it was discovered in traces. Belady’s 1966 study compared page-replacement policies by replaying real program traces through a simulator, and Denning turned the working set into a measurable object and spent decades documenting locality as an empirical regularity of workloads, not a law. Caching became engineering the day it acquired instruments — traces, hit-rate curves, reuse distances. Every replacement policy since has been a bet about measured workload structure.

Agent memory is making the same shape of bet. Michie introduced memoization in Nature in 1968 as “rote learning” — remember results instead of recomputing them — and a persistent agent store is the semantic version of that wager, as I argued in More Than a Cache. What the bet does not yet have is instruments. The field is roughly where caching was in 1965: real systems, strong intuitions, and no workload science.

Exhibit A: we cannot currently measure a hit #

If you ask whether an agent memory system works, the number you get quoted almost certainly comes from LoCoMo, a benchmark of long synthetic conversations. Look at that instrument up close. A 2026 audit by Penfield Labs found that 6.4% of the answer key is wrong — 99 of 1,540 questions carry hallucinated facts, bad temporal reasoning, or misattributed speakers — and, worse, that the LLM judge doing the scoring accepts up to 63% of intentionally wrong answers. One analysis reports that the standard evaluation harness silently drops the 446 adversarial questions — 22.5% of the benchmark — whose correct answer is “that isn’t in the history,” and instructs models never to abstain.

On this substrate, the vendors fight. Zep’s headline 84% fell to 58.44% under a corrected re-run filed by Mem0’s CTO; Mem0’s paper had scored Zep at 65.99%, and Zep’s own re-run put it at 75.14%; MIRIX reports 85.4% on the same benchmark. Published numbers for competing systems span 58–85%, on an instrument whose judge noise is wider than any difference being claimed. A recent survey of the evaluation landscape, Anatomy of Agentic Memory, documents the pattern systematically: benchmark saturation, judge sensitivity, backbone-dependence, and system costs that go unreported.

I don’t read this as scandal. I read it as a diagnostic of the field’s developmental stage. An evaluation is a measurement instrument, and — as I argued in an earlier post on evaluation resolution — an instrument whose error bars exceed the effect size measures nothing. Ranking memory architectures with today’s benchmarks is like comparing replacement policies before anyone had a trace.

The decomposition I want instrumented #

Here is the research program, stated as an accounting identity:

In words: a memory architecture’s value is the sum of separable decisions — what gets written at all, in what form, arranged how, found how, compressed when, distrusted when, and discarded when — and each term should be measurable on its own, because each fails differently.

I want to be precise about what already exists, because the field is further along on some terms than the benchmark chaos suggests. The read path is partially instrumented: LongMemEval explicitly decomposes memory into indexing, retrieval, and reading stages, and shows that stage-level optimizations each measurably improve recall and downstream accuracy. And the representation term has one genuinely good experiment: On the Structural Memory of LLM Agents compares chunks, knowledge triples, atomic facts, and summaries across tasks and finds that different structures win on different tasks, with mixed memory most robust to noise. That is the most direct published support for what I take to be the central conjecture: memory architecture value is workload-dependent, and there is no universally best design — only designs matched or mismatched to reuse patterns.

What has essentially no per-term instrumentation is everything on the write and maintenance side. No benchmark scores the memory artifact itself — admission precision, consolidation loss, update correctness — independently of downstream question-answering; most don’t score write cost at all. (The admission term deserves its own essay, and gets one: The Write Path.) Invalidation and eviction are measured nowhere I can find.

And the deepest gap sits underneath all seven terms: nobody has measured the locality structure of cognitive workloads. Denning’s working-set methodology — reuse-distance distributions extracted from real traces — has, as far as my searching can establish, never been applied to agent workloads. We do not know the re-reference rate of knowledge items in real deployments, the shape of the semantic hit-rate curve, or what the equivalent of cache pollution looks like when the polluting entry is a plausible-sounding wrong synthesis. Until someone builds those traces, every admission and eviction policy in every shipping memory product is a guess about a distribution no one has plotted.

The first data points, held loosely #

A few early measurements exist, and I want to cite them the way early data deserves — as first observations, not established phenomena. A July 2026 study of filesystem-based agent memory found that organization roughly halves retrieval cost at scale but that “no agent we measure converts organization itself into better answers” — one paper, one setting, and the subject of its own post in this series. A July 2026 longitudinal preprint found that memory-architecture rankings invert with history length — a curated-map memory at 96% recall after three weeks fell to 72% by nine weeks while a provenance-typed graph rose to ~90% — though I hold this one especially loosely: it is unreplicated, appears to be a single-author preprint, and runs on fully synthetic “life-script” data. Mem0’s own evaluation shows its memory system cutting p95 latency by 91% and tokens by ~90% versus stuffing the full history in context — while, as Zep’s reading of Mem0’s own table points out, the full-context baseline still wins on accuracy (~73% vs ~68%). And sleep-time compute gives the cleanest economics of memoized cognition so far: precomputing understanding before queries arrive cuts test-time compute about 5× at equal accuracy, with the benefit governed by how predictable the queries are — reuse probability, again.

That is close to the entire empirical inventory: thin, weeks-to-months old, partly vendor-reported, and standing on a benchmark substrate we know is cracked. MemoryAgentBench finds that no current system masters even its four basic competencies. None of this embarrasses the field; it locates it. Caching looked exactly like this before trace-driven simulation existed — plausible mechanisms, contradictory anecdotes, no shared instruments — and it became one of the most quantitatively successful subfields in systems within a decade of getting them. The questions here are just as concrete and just as answerable: what should be written, what should stay raw, when does hierarchy pay, how does performance change as history grows from days to years, what is semantic cache pollution, what is the locality of thought-work. Someone has to go build the traces.

What would change my mind: A LoCoMo successor with an audited answer key, scored abstention, and validated judging that reproduces today’s leaderboard rankings would convince me the current instruments were adequate after all and the “pre-empirical” framing overwrought. And a trace study showing agent workloads have no exploitable locality — flat reuse-distance curves over long-horizon deployments — would tell me there is no science of semantic memoization to build, because memoization without locality is just storage.