---
title: Day 6 · The Cheapest Tokens Are the Ones You Never Retrieve
type: newsletter
date: 2026-05-28
source: linkedin
summary: Most production prompts are not written. They are assembled. A user asks a question. Before the model sees anything, a retrieval layer runs — pulling rows from a database, passages from a document store, results from a search index, nodes from a knowledge…
newsletter: Green AI Bytes
draft: false
---

Most production prompts are not written. They are assembled.

A user asks a question. Before the model sees anything, a retrieval layer runs — pulling rows from a database, passages from a document store, results from a search index, nodes from a knowledge graph, vectors from a similarity store, conversation history from a session log, sometimes data from a live API. Whatever the underlying system, the pattern is the same: the model does not have your enterprise data on board, so something has to fetch it, shape it, and stitch it onto the prompt before the call goes out.

In most real systems, the retrieved content is much larger than the instructions, the examples, and the user's question combined. Days 3 through 5 optimised the prompt as if it arrived fully formed. It does not. The layer that builds the prompt is its own efficiency frontier, and for many enterprise teams it is the largest one in the stack.

A customer-support assistant is built on a retrieval pipeline. A user asks: *"How do I cancel my subscription?"*

The pipeline fires. In this team's case it uses a vector store, but the same shape of waste appears whether the backend is a search index, a graph, a SQL layer, or a hybrid of several. The system retrieves the top twenty candidates by similarity, each roughly four hundred tokens long, and concatenates all of them into the prompt. The system prompt adds another six hundred. The user's question is fifteen tokens.

The model receives an eight-thousand-six-hundred-token prompt to answer a question whose actual source material is one paragraph in the cancellation policy document.

The team measured. Of the twenty retrieved passages, the answer lived in passage three. Passages one and two were about pausing, not cancelling. Passages four through twenty were about refunds, billing cycles, account deletion, and re-activation — related topics that the retrieval layer scored as similar, but that contained no information the model used.

The team made three changes. They cut top-k from twenty to five. They added a re-ranking step that scored each candidate against the actual question, not just the initial retrieval signal. They reduced passage size from four hundred tokens to one hundred and fifty, with a small overlap.

The prompt dropped to one thousand four hundred tokens. Answer quality, measured on their eval set, went up — not down — because the model was no longer working past sixteen near-misses to find the one passage that mattered.

Three patterns produce this kind of bloat across every retrieval method, and all three feel like good defaults:

**The top-k default.** Every retrieval system ships with a default — twenty candidates, ten results, the first page of matches. Most teams never change it. Defaults are set for demos, not for production.

**The chunk-size inheritance.** Most retrieval tutorials use a token count from the example they were copied from. Teams inherit the number without asking whether their content has that shape.

**The "context is free" reflex.** Million-token context windows have made teams stop pruning what they retrieve. The window can hold it, so the pipeline sends it. Every token in that window is still billed, processed, and consumes compute.

The model sees what retrieval gives it. The retrieval layer decides the size of the bill before the model sees a single token.

### The long-context question

A fair question to put against this principle: with context windows now stretching to a million tokens and beyond, is the retrieval layer still needed at all? The case is sometimes made that you can simply send the whole corpus on every query and let the model find what matters.

The technical answer is that long context and retrieval are not substitutes — they are different tools that solve overlapping problems on different cost curves.

Long context genuinely helps. Models have meaningfully improved at finding relevant information across long inputs, and some workloads that needed careful retrieval two years ago — analysing a single long document, working through a moderate codebase, holding a long conversation — now run cleanly inside the window without it.

But three things have not changed. Every token in that window is still billed, still processed, and still carries an infrastructure and emissions cost. A million-token call can cost orders of magnitude more than a one-thousand-token call, and often adds materially higher latency. Recall across long contexts is also uneven — strong at the beginning and end, weaker in the middle, weaker still when the question requires connecting information from multiple places. And corpora that exceed the window, content that changes daily, and queries that need fresh enterprise data still require a retrieval layer regardless of how large the window grows.

The result is not *retrieval or long context.* It is *retrieval and long context, used for what each is best at.* Long context handles the working set in front of the model. Retrieval handles the larger enterprise world the working set was selected from. The efficiency principle below applies to both — the cheapest token, whether it came from a retrieved passage or a window the team chose to fill, is the one the system never had to send.

The operating principle in the retrieval layer is a four-step hierarchy:

**Filter → Rank → Right-size → Refresh.**

*Filter* before you retrieve. Most queries can be narrowed before any retrieval method runs — by metadata, by user permissions, by product area, by language, by recency, by document type, by the natural partitions in your data. A pre-filter that narrows the candidate set before retrieval runs costs almost nothing, and the smaller the set it hands on, the faster and more accurate whatever comes next becomes. *Rank* what survives. The first-pass score from any retrieval system — similarity, keyword match, graph distance — is a coarse signal. A second-pass ranker that scores candidates against the actual question catches the result that was relevant but not surface-similar. *Right-size* what reaches the model — top-k is a knob, not a default, and the right value is almost always smaller than the one shipped in the example you copied from. Chunk size, where it applies, is the same — match it to the natural shape of an answer in your content, not to a number from someone else's setup. *Refresh* what the system holds. Caches grow stale, indexes drift as content updates, embeddings drift as the underlying model updates, and a stale layer returns wrong answers that the model then phrases convincingly.

The first rung is the most often skipped. Teams reach for retrieval tuning before they have asked whether a metadata filter, a permission check, or a SQL WHERE clause would have cut the candidate set in half. Pre-filtering is the cheapest, most effective form of retrieval optimisation, and it is the one nobody talks about because it does not involve a model.

The third rung is the most often mis-set. Top-k and chunk size are usually inherited from a tutorial and never re-measured. Both should be tuned to the eval set, not to a screenshot from someone else's blog post.

**Engineering.** Measure two numbers per retrieval call: how many items you retrieve, and how many of them materially contributed to the final answer. The gap between those two is your retrieval waste. Tune top-k and chunk size to your eval set, not to defaults. Add a re-ranker before you tune anything else — it is the single highest-leverage change most retrieval systems have not made.

**Platform / Infrastructure.** The retrieval layer has its own cost curve — index storage, query compute, latency, and the per-token cost it adds to every downstream model call. Track retrieved tokens as a first-class metric alongside prompt tokens and output tokens. If you only see two of the three, the largest one is hiding.

**Architecture / CTO.** Retrieval is a system, not a library call. It deserves the same design discipline as the model layer — ownership, versioning, eval coverage, and a clear contract for what it returns. Many enterprise retrieval pipelines were assembled quickly during early experimentation and have not been systematically revisited since. The choice of method — vector, keyword, graph, hybrid, federated — is itself a design decision and worth revisiting as the data and the workload evolve.

**Sustainability / ESG.** Retrieved tokens consume compute in the same way as written prompt tokens, and should be measured as part of the system's overall AI workload. A system that retrieves ten thousand tokens to answer with a hundred has a tokens-per-task ratio that no amount of prompt pruning will fix.

**Business / Product.** The quality of an AI answer is downstream of what the retrieval layer surfaces. A wrong answer is rarely the model inventing from nothing — it is the model accurately answering using the wrong source. Better retrieval is often the cheapest product improvement available.

Five seats, one ladder, one retrieval layer that earns its tokens.

Pick one feature this week that uses retrieval — of any kind. The one with the longest prompts, or the one whose answers most often feel slightly off-topic.

Pull twenty real queries. For each one, capture what the retrieval layer surfaced, and record two numbers: how many items were retrieved, and how many of them materially contributed to the final answer.

The ratio of *useful items* to *retrieved items* is your **retrieval** **signal-to-noise**. In most untuned systems, it is under twenty percent. Every retrieved item that did not contribute to the answer was paid for in tokens, latency, and energy.

Walk the ladder. *Filter* by metadata, permissions, or partition before retrieval — start with the most obvious filter your data allows. *Rank* the survivors with a second-pass scorer. *Right-size* top-k and chunk size against your eval set, not against the defaults. *Refresh* the cache and re-index any content that has changed since the last build.

One feature, one ladder, two ratios, twenty queries. That is the work.

> *The cheapest token is the one the retrieval layer never fetched. The second cheapest is the one it fetched because the answer actually needed it.*

Filter, rank, right-size, refresh — that is the work in the layer that feeds the call. But the call also has a return trip. The model generates an answer, the system parses it, an application acts on it, sometimes a second call is made to clean up the first. Most teams pay for the same answer two or three times because what came back was not in the shape they could use. The next principle is what efficient AI looks like in the layer that catches the response. See you in the next issue.

---