Day 4 · Make the Call Count Smaller
The cheapest, fastest, lowest-carbon inference is the one the system never made. Providers already reduce some waste below the API line through prefix caching. But the biggest opportunity lives above it — in calls the application never needed to make in the first place.
A customer support assistant answers what are your business hours forty thousand times a day. Each answer is generated fresh. Same question. Same context. Same answer. Forty thousand fresh inferences.
The model is doing exactly what it was asked to do. The system around it is the problem. It has no memory that this question has been answered before, no instinct to check before spending.
Add a lightweight cache in front of that endpoint, keyed on the normalised question, and the economics can change immediately. The model is no longer generating the same answer thousands of times a day. Spend falls because repetition disappears.
For many production AI systems, this becomes one of the largest recoverable pools of waste after model right-sizing. And unlike right-sizing, it usually requires no model change at all.
Three quiet forces keep the call count high. The first is the provider-cache illusion — teams know major providers already cache prompt prefixes, assume the work is done, and stop looking. The second is the fresh-is-safer reflex — every call regenerated from scratch feels more correct, so nothing gets cached even when the answer is provably stable. The third is the agent multiplier — one user query becomes ten model calls inside a loop, and no one is counting the eight that were redundant.
The result is inference running at scale on questions already answered, documents already fetched, and retries that should never have reached the model.
Major providers already reduce repeated prompt work through prefix caching when prompts stay stable. Most teams benefit from those savings automatically — often without even realising it.
That is the floor. The ceiling lives somewhere else.
Provider caching only sees what crosses the API. It cannot see the duplicate call your application made on behalf of two different users. It cannot see the document your agent re-fetched in the next turn. It cannot see the retry that should have caught a network blip three layers below the model.
The operating principle above the API line is a four-step hierarchy:
Dedupe → Reuse → Carry → Catch.
Dedupe identical and near-identical calls before they leave the application. Reuse what the provider already cached by keeping the prefix stable instead of mutating it. Carry context forward through an agent loop instead of rediscovering it each step. Catch failures at the layer they happened on, so the model never retries what the network owns.
Every other call-reduction technique sits somewhere on this ladder.
The most overlooked rung is the third one. Agent loops are where call counts quietly explode — one user query, ten model calls, half of them re-reading something earlier in the loop already had. What you carry forward, you do not pay for again.
Engineering. Walk the ladder in order. For one week, log every question your system sends to the model and count how many are repeats. Most systems have a small set of questions that show up over and over again — put a simple cache in front of those and the call count drops the day you ship it. Then look at your agents: each step in a loop often re-fetches or re-summarises something an earlier step already had. Carry that forward instead. Finally, look at where the model is retrying on errors that have nothing to do with the model — a flaky network, a malformed response — and move those retries to the layer that actually owns the failure. Each of these pays off the day it ships.
Platform / Infrastructure. The team that runs the model gateway is the only one that sees the full traffic shape across the organisation. A shared cache, sitting in front of the model and used by every team, plus a simple dashboard showing how often calls repeat — that is the cheapest visibility investment a platform team can make this quarter. Build it once, well, so every team in the org does not build a worse version six times over.
Architecture / CTO. Treat the cache as part of the architecture, not as a switch each team flips on their own service. Two design rules carry most of the weight. First, keep the unchanging parts of your prompts in the same place every time — the providers already cache that for you, but only when it stays stable. Second, when two users ask the same question in different words, the system should recognise that and return the same answer. Both of these are decisions made once at the architecture level and paid back on every call after.
Sustainability / ESG. The number missing from most dashboards is the calls that did not happen. Every metric we track today — cost per call, energy per call, carbon per call — assumes the call took place. None of them count the ones the system avoided. Add avoided calls as a first-class number alongside the unit metrics, and the work that moves it suddenly becomes the work that gets funded.
Business. The cost of inference is not the price of a call. It is the price of a call multiplied by the number of calls. The first number gets negotiated in contracts. The second is where the real money lives — and in most untuned systems, a meaningful share of that volume is the same question, asked again, paid for again. That share, not the unit price, is where the spend hides.
Five seats, one ladder, one bill that moves.
One thing to do this week: pick one AI feature you have in production. For seven days, keep a record of every question that feature sends to the model.
Then ask two questions of that record.
How often is the same question being asked more than once? And how much of the total traffic comes from a small handful of repeat questions?
In most systems, the answers are uncomfortable. A surprising share of the work the model is doing is work it already did.
Put a small cache in front of the questions that repeat the most. Send the answer from memory the second time, the third time, the four-thousandth time. Then measure three things before and after the change — how many calls the feature made, what it cost, how fast it felt to the user. Share those three numbers with the five seats from The Lens.
That is the experiment. The numbers tell the rest of the story.
The cheapest call is the one you never make. The second cheapest is the one you make once and remember.
Coming up next. Prune, compress, place, cap — that is the work inside a single call. But every prompt is built from something before it is sent. Retrieved passages, database rows, prior conversation, search results, knowledge base chunks. In most production systems, the prompt the model sees is mostly retrieved, not written — and the cheapest tokens in that prompt are the ones the retrieval layer never sent in the first place. The next principle is what efficient AI looks like in the layer that feeds the call. See you in the next issue.