More Than a Cache

The cache analogy for LLM knowledge bases seduces because part of it is literally on the price sheet. Anthropic charges a 25% premium to write a prompt prefix into cache and 10% of the base rate to read it back (prompt caching); OpenAI discounts recently seen tokens by half. Persistence has a write premium and a read discount — the memoization tradeoff, priced by the API. And the frame has pedigree. Karpathy sketched the LLM OS in November 2023 with “RAM: 128Ktok” on the spec sheet; MemGPT had engineered “virtual context management” on the operating-systems analogy a month before that; GPTCache shipped a semantic cache for LLM responses the same year. So when the personal LLM wiki pattern went viral this spring, the slot was already labeled: the context window is RAM, the wiki is one more level of cache — farther away, bigger, slower.

I have used the analogy that way myself, and it earns its seduction. But it fails in three specific places, and the failures are more instructive than the fit. The tell is hiding in plain sight, in Karpathy’s own choice of verb.

Caches store what they saw; wikis store what they ran #

In the tweet that launched the pattern, Karpathy doesn’t say cache. He says he uses an LLM to incrementally “compile” a wiki — his quotation marks, his word. That is a different claim about what the stored object is. A CPU cache holds bytes it has seen, keyed by the address it saw them at. A wiki page holds the result of computation the system ran: many papers went in; retrieval, comparison, terminology reconciliation, and synthesis happened; one conceptual model came out. The gist calls the artifact “compounding,” and that is the right word — nothing in a cache compounds.

This reframing has honest ancestors, and older ones than MemGPT. Donald Michie introduced “memo” functions in Nature in 1968 — the paper that coined memoization — and framed the technique as rote learning: a machine that improves by keeping the results of computations it has already done. Storing cognition, not bytes, has been on the table for fifty-eight years. The closest database relative is the materialized view — an expensive query result persisted so it never has to be recomputed — and the point of that literature is that the persistence is the easy half; the maintenance problem spawned a research field. And a compiled wiki resembles a compiler intermediate representation: a representation chosen to serve future computation, not to mirror the source.

The economics of memoized cognition are even starting to be measured. Sleep-time compute — reasoning over a context before the questions arrive — cuts test-time compute roughly 5× at equal accuracy, and there is now work on caching intermediate summaries rather than final answers. The valuable stored object is not a copy of the source material. It is the residue of expensive thinking.

Every row above the last has a working answer in its final column. That empty cell is where the analogy breaks.

Three places the cache analogy fails #

A hit has no ground truth. A cache hit is exact and binary: the address matches or it doesn’t, and the bytes returned are the bytes stored. A wiki “hit” is a judgment — is this page about what I’m asking, and does it still contain what I need? — made over a lossy transformation. Even a correct lookup can return a synthesis that discarded the one detail today’s question turns on, and there is now a small literature just on calibrating when a semantic-cache lookup should count as a hit at all. This is the deeper reason writing knowledge down doesn’t make retrieval disappear — it changes what retrieval means — which is the argument of the previous post in this series.

Semantic invalidation is unsolved, and it is twinned with naming. Phil Karlton’s line — “there are only two hard things in Computer Science: cache invalidation and naming things” — reads like a joke until you notice an LLM wiki has both problems at once, because its page names are its address space. Hardware coherence has protocols; a bus snoop announces every write. But a page synthesized from twelve sources goes stale when the world changes in ways no signal announces, and no protocol exists for detecting that. Even the freshest standardization attempt concedes the point: Google’s Open Knowledge Format — a v0.x spec only weeks old, so an indicator rather than an institution — added a stale_after date field in v0.2, reducing staleness to a calendar comparison. That is less a solution than an honest admission that nothing better is agreed. What real invalidation would require — a store that maintains a live relationship with the changing world — is a big enough problem that I give it its own post.

The compile step can lie. SRAM does not editorialize. A cache can serve stale bytes, but it never invents them. An LLM performing synthesis can hallucinate, and a hallucination written into the wiki becomes something a transient error never was: a persistent, retrievable, trusted mistake, wearing the store’s own authority. There is early evidence this failure mode compounds rather than washes out: agents measurably follow bad stored records once retrieved, and a July 2026 preprint on filesystem memory — one study, unreplicated, so a first observation rather than a settled result — found that an agent given a free reorganization pass silently condenses content away unless given an explicit preservation rule. Corruption at write time is exactly why I think the write path deserves equal billing with retrieval.

A hierarchy of borrowed problems #

So the right analogy is not “wiki = cache.” It is the whole semantic storage hierarchy at once — cache and materialized view and compiler IR and search index — each a partial fit, and each, more usefully, importing its home field’s research agenda. Cache theory contributes admission and eviction under locality, along with Belady’s warning that optimal eviction requires knowing the future. Materialized views contribute incremental maintenance. Compiler IRs contribute the discipline of representations chosen for future computation rather than human reading. Indexes contribute findability. I claim no novelty for any single mapping — MemGPT, GPTCache, and the OS framing all predate the wiki pattern by years — only for the sorting: which analogy carries which load, and where each one snaps.

Because what none of the borrowed fields provides is a theory of stores whose contents are lossy beliefs about a moving world, written by a process that can err. The analogies mark the boundary of what we can import. Past that boundary, the work is genuinely new — and each of the three breaks above is, I think, a research program rather than a footnote.

What would change my mind: A demonstrated semantic-invalidation mechanism — a system shown at scale to detect when a synthesized page’s premises have changed and to repair it with measured correctness — would collapse my central distinction, making the wiki just a cache with a slower coherence protocol. Likewise, longitudinal evidence that hallucinated writes get diluted or corrected by ordinary use, rather than trusted and propagated, would erase the third break and leave the cache analogy in far better health than I’ve granted it here.