--- title: Day 5 · Spend Tokens Like They Cost Money type: newsletter date: 2026-05-26 source: linkedin summary: Published on --- A surprising amount of AI waste hides inside calls you already decided to make. Most production prompts are two to ten times longer than necessary — packed with instructions the model already knows, examples that no longer matter, and context… newsletter: Green AI Bytes draft: false --- Published on --- A surprising amount of AI waste hides inside calls you already decided to make. Most production prompts are two to ten times longer than necessary — packed with instructions the model already knows, examples that no longer matter, and context left over from three turns ago. The model still runs. The waste simply rides inside the call. A document-summarisation feature ships with a 2,400-token system prompt. Eleven role instructions. Six tone guidelines. Four formatting rules. Three negative examples. A persona paragraph. A safety preamble copied from a tutorial nine months ago. Then the document — usually under 800 tokens — gets appended at the bottom. The team removed one section at a time and re-ran their eval set. The persona paragraph changed nothing. Four of the six tone guidelines were already covered by the model's defaults. Two of the three negative examples were redundant. The safety preamble was duplicating system-level guardrails the provider had already applied. The cleaned prompt was 380 tokens. Quality scores held within noise. Latency dropped by a third. The bill on that one endpoint fell by more than seventy percent — and the model was never changed. This is the third recoverable pool of waste in the stack, after right-sizing the model and reducing the call count. It is the largest one that lives *inside* a call you have already decided to make. Three quiet forces keep prompts bloated. The *more-instruction reflex* — when a model gets something wrong, the instinct is to add another rule rather than test whether an existing one is doing any work. The *example accumulation drift* — few-shot examples are added during prototyping and never pruned, because nobody is sure which one is load-bearing. The *context-window illusion* — million-token windows make tokens feel free, even though every one of them is billed, processed, and burned. Prompts only ever accumulate. The operating principle inside a call is a four-step hierarchy: **Prune → Compress → Place → Cap.** *Prune* every instruction, example, and context segment that does not change an evaluation outcome. *Compress* what remains into the tightest form the model still understands — shorter phrasings, fewer examples, references instead of restatements. *Place* the stable content first so prefix caches keep working, and the variable content last so each call only pays for what changed. *Cap* the output by telling the model exactly how long the answer should be, rather than letting it run to the model's natural length. The first rung is the most skipped. Teams jump straight to compression — shortening words, asking the model to "be concise" — without first asking whether the instruction needs to exist at all. A deleted instruction has zero tokens and zero cognitive load on the model. Compression only optimises what survived a question that was never asked. The fourth rung matters more than it looks. On many provider price lists, output tokens are materially more expensive than input tokens, and they are generated sequentially — which means they drive latency and energy per request as well. A prompt that produces a 600-token answer when 150 would do is leaking on every dimension at once. **Engineering.** Take your top three prompts by traffic. Remove one section at a time and re-run your eval set. Keep what moves quality. Delete what does not. Most teams find at least one entire block — a persona, a tone instruction, a redundant example — that contributes nothing. Then cap your max\_tokens to the actual P95 length of useful answers, not to whatever default the SDK shipped with. **Platform / Infrastructure.** Track input and output tokens by prompt template. As a rough rule of thumb, anything where the system prompt is more than three or four times the typical user content is worth a closer look — though the right ratio varies by workload. **Architecture / CTO.** Treat prompts like code: version, review, and test them. Prompts without ownership only grow. **Sustainability / ESG.** Tokens-per-task is one of the clearest application-layer proxies for energy efficiency. Trend it release over release. **Business / Product.** Output length is a product decision. A 600-word answer when the user wanted a sentence is both a UX bug and a cost bug. Five seats, one ladder, one bill that moves. Pick one feature this week. The one you suspect has the most bloated prompt — usually the oldest one, or the one with the most contributors. Write down three numbers before you change anything: the current token count of the system prompt, the average tokens-per-response, and the cost per thousand calls. Then walk the ladder. Prune one section. Run the eval. If quality holds, prune the next. Compress what survived. Cap the output to the actual useful length. Re-measure. One feature, one ladder, three numbers. That is the work. *The cheapest token is the one you never wrote into the prompt. The second cheapest is the one the model never had to generate back.* Prune, compress, place, cap — that is the work inside a single call. But most AI systems today are not single calls. They are loops. An agent receives a query, plans, calls a tool, reads the result, decides again, calls another tool, and so on — sometimes ten or twenty turns deep. Every turn carries a fresh prompt, a fresh context, and a fresh bill. The next principle is what efficient AI looks like once the unit of work is no longer a call, but a loop. See you in the next issue. ---