New
articleMigrating a 20-Year WordPress Blog to Be Agent-Readytechnology bytesAGI Has Arrived. Now Give It a Simpler Problem.podcastEpisode 100: The Next Era of AI — From Agents to Enterprise IntelligencearticleMigrating a 20-Year WordPress Blog to Be Agent-Readytechnology bytesAGI Has Arrived. Now Give It a Simpler Problem.podcastEpisode 100: The Next Era of AI — From Agents to Enterprise Intelligence
Navveen Balani Making systems intelligent, efficient & responsible

Day 12 · Cache What Doesn't Change

View as markdown

Days 8–11 worked the loop itself — its turn count, its route, its tool shape, its context weight. That closes the loop chapter. This issue opens a new one. The next three days work on the platform underneath the loop, so the same work costs less every time it runs.

Here is the observation that opens it. Most AI prompts have a shape people rarely look at. Depending on the workload, anywhere from forty to ninety-five percent of the prompt is the same on every call — the system prompt, the tool schemas, the persona, the document the assistant is reading. The remainder is the user’s actual question. The model processes the whole thing from scratch every time, as if it had never seen any of it before.

Every cached prefix is a forward pass the system does not have to run again. Every uncached prefix is one it does, on every call, forever. The difference is not in the model. It is in whether the platform treats the prompt as a string to reprocess or a structure with a stable part to remember. Caching is the discipline of refusing to recompute what has not changed — and it is often one of the largest cost-and-carbon levers in production AI systems, partly because it asks engineers to look at prompts as data, not as text.

Two support assistants, same hour, same thousand customer questions. Both run the same model. Both return the same answers. One reprocesses the system prompt, the tool schemas, and the policy document on every single call. The other declares those once, and pays for them once. By the end of the hour, the second assistant has run at a fraction of the cost, at a fraction of the latency, and at a fraction of the energy. The model did not change. The model’s bill did.

Three patterns surface across almost every production AI workload, and each one is a place where the same tokens are processed again and again.

The Stable Prefix. Assistant identity, tool schemas, policy rules, document context — unchanged, yet recomputed every call.

The Repeated Question. Thousands of users asking variants of the same support question, each one triggering a full inference path.

The Reloaded Document. The same hundred-page PDF chunked, embedded, and re-attended across turns as if the system has never seen it before.

Most production AI does not have an intelligence problem. It has a memory problem — it forgets what it just computed.

“But our prompts are all different”

The most common objection to caching is that no two calls look alike, so there is nothing to reuse. Look at the actual prompt structure. The variation is almost always in the suffix — the user turn, the latest tool result. The prefix is system prompt, tools, persona, retrieved context, conversation history. Research measuring shared-prefix rates across real production traffic (Yuan et al., DualMap, arXiv 2026) found that roughly three in four agent-and-tool requests share more than half of their prompt with prior requests, and nearly half of conversational requests do the same. In long-context production workloads, that prefix routinely makes up the large majority of the tokens. The illusion of uniqueness comes from looking only at the tail.

There is a four-rung ladder for moving from a naive prompting pattern to a cache-aware system. Each rung is a small change in how the prompt is built, not a rewrite of the application.

Locate. Open one of your typical prompts and mark which bytes are the same on the next call and which bytes are not. The stable region — system prompt, tools, document context, conversation prefix — is the cacheable surface. If you have never done this exercise, the first reaction is usually surprise at how much of the prompt qualifies.

Pin. Tell the provider which region is stable. Most leading model providers now expose some form of prompt or prefix caching — a marker that says this prefix will repeat; remember the computation for it. Cached tokens typically bill at a fraction of the normal input cost on subsequent calls (Anthropic’s prompt caching documentation and the AWS Bedrock caching guide both report up to ninety percent input-cost reduction and up to eighty-five percent latency reduction on cached portions of long prompts), and they reduce the repeated prefill work that drives a significant share of per-call energy on long-prompt workloads. Pinning is the move from accidental reuse to declared reuse — and on a high-volume endpoint, declared reuse is the difference between paying for the same prefix once a day and paying for it ten thousand times.

Layer. Prompt caching handles repeated prefixes. It does not handle a different user asking the same question through a different prefix. For that, add a semantic cache above the model — a small lookup that recognises when a new query is close enough to a previously answered one to return the stored answer directly. The right similarity threshold is application-specific, but the principle is universal: the cheapest inference is the one that never reaches the model.

Watch. Cache hit rate belongs next to latency and cost on the dashboard, not in a quarterly review. Published production data on Claude Code shows sustained cache hit rates around ninety percent in real coding sessions — a useful upper bound for what a well-structured agentic workload can achieve, and a useful yardstick for what most teams are leaving on the table today. A system whose hit rate is drifting downwards is a system whose prompts are drifting — usually because someone added a timestamp, a session ID, or a randomised greeting into the stable region and broke the prefix match. Make the metric visible, and the breakage gets caught in days instead of months.

Engineering. Restructure prompts so the stable part comes first and the variable part comes last. Cache markers attach to prefixes; a single dynamic token at the top of the prompt invalidates everything below it. Prompt order is now a performance decision, not a styling one.

Platform and Infrastructure. Serving stack matters. Provider-side caching works for hosted models. For self-hosted inference, engines such as vLLM and SGLang offer prefix caching across requests, with knobs for TTL, eviction policy, and cross-tenant isolation. The platform team owns those dials, and exposes a semantic-cache layer as a shared service so every product team is not building its own.

Architecture and CTO. Caching is not a post-hoc optimisation. It is a prompt design constraint, decided upstream. Decisions about retrieval strategy, conversation history format, and tool schema stability all flow into whether the system can cache at all. An architecture that treats each call as fresh leaves one of the largest efficiency levers permanently disabled.

Sustainability and ESG. Cached prefixes reduce or avoid repeated prefill computation, which is where a significant share of per-call energy is spent on long-prompt workloads. A high cache hit rate translates directly into fewer GPU-seconds per business outcome. This is one of the clearest cases where the cost metric and the carbon metric move in the same direction, on the same lever, with the same intervention.

Business and Product. Cost on the cached portion typically drops by a large multiple versus uncached input. Latency on a cache hit can fall from seconds to milliseconds. For high-volume features — assistants, document Q&A, copilots — the difference between caching and not caching is often the difference between viable unit economics and unviable ones.

Five seats, one prompt structure that respects what has already been computed.

The work is to find the bytes that repeat, declare them, layer a semantic cache above them, and watch the hit rate as a living metric. Pick one high-volume endpoint. Measure its current cache hit rate — most teams discover it is zero. Restructure the prompt so the stable prefix is contiguous and comes first. Turn on provider caching. Add the hit rate to the dashboard the same week.

One caution, so you do not over-correct. Not every call benefits from caching. Low-volume endpoints, one-off analyses, and prompts that genuinely change top to bottom on every call are not the target. Caching pays where the same prefix runs again and again. Find those endpoints first, leave the long tail alone.

One prefix, one marker, one hit-rate metric. That is the work.

An uncached prompt pays for the same thinking on every call. A cached prompt pays for it once.

Caching deals with the work you have already done. The next move is about the work you do together — why one well-shaped call can be cheaper, faster, and greener than many small ones, and what batching changes about the economics of inference. See you in the next issue.