Day 17 — The Invisible Payload
Day 16 traced the compute that safety adds on the outside of every call. Day 17 opens the same call up and looks at what is actually inside it — the request the model reads, not the message the user sent.
The gap between the two is usually enormous. A user types a short question. The model receives a payload many times larger: a system prompt written months ago, a tool catalog listing every capability the agent might ever need, a set of example exchanges added to fix behaviours that are now stable, formatting instructions, safety framing, retrieved context, and the running history of the conversation. All of it assembled by the framework. All of it billed as input tokens. All of it re-sent on the next call.
This overhead is not wrong — much of it is what makes the model behave correctly. It is invisible, though, and invisibility is where waste hides. The point of today’s issue is to make the assembled request visible and treat its composition as a first-class engineering surface.
A product team ships a customer support agent. A user asks: “How do I reset my password?” — seven words, roughly ten tokens. A trace of the outbound request shows the model actually receives thousands of tokens. The user’s question is buried near the end.
Three patterns recur across teams that open their assembled requests for the first time.
The kitchen-sink system prompt. Every rule ever added is still shipping. Persona description, formatting guidelines, tone-of-voice notes, brand-voice do’s and don’ts, an evolving list of edge cases with example handling, and safety framings added after specific incidents. Each addition made sense at the time. None of them were removed when the reason passed. The prompt has become a changelog no one reads and every call pays for.
The everywhere tool catalog. All available tools are declared on every call, even calls where none of them will be used. Every tool description, every parameter, every enum value ships as input. On a route where the model is answering a simple FAQ, the schemas for booking, refund, escalation, and account management still ship in full, because the framework does not know which are relevant.
The few-shot fossils. Example exchanges added early to teach a specific behaviour remain in the prompt long after the behaviour has been stabilised through other means — better instructions, a fine-tune, a schema constraint. The examples remain because no one is certain what behaviour they still support.
The prompt the model reads is not the prompt the user wrote. And the difference between the two is what shows up on the bill.
A reasonable objection: these instructions are what make the model behave correctly, and cutting them risks quality. Fair. This is not an argument for trimming on faith. It is an argument for making the assembled request visible, testing each component against the behaviour it earns, and moving the stable parts to surfaces that do not charge for them on every call. Nothing is removed without knowing what it was doing.
Four rungs turn an opaque assembled request into a shaped, observable payload. Each rung asks a different question about what is in the prompt and where it should live.
Inventory. Print the full outbound request end to end, including every wrapper the framework adds silently. Count the tokens contributed by each component — system prompt, tool schemas, examples, formatting rules, retrieved context, conversation history, the user’s actual message. Most teams have never seen this breakdown. The audit alone often surfaces the largest lever in the system.
Trim. For each component, ask: does this earn its cost on this call? Formatting rules the model reliably follows without prompting. Few-shot examples fixing behaviours the fine-tune now covers. Tone-of-voice guidance duplicated across three sections. The test is not whether the content is useful — it is whether removing it changes the output. Trim what does not.
Split. Separate the stable spine of the request from the volatile head. The spine — system prompt, tool schemas, canonical examples, policy framing — changes rarely, if ever. The head — the user’s query, current retrieved context, session state — changes every call. Today they are concatenated and shipped together. They do not have to be.
Amortise. Route the stable spine through surfaces that charge once, not per call. Provider-side prompt caching is one path. Prefix sharing, distillation into a fine-tune, or configuration lifted out of the prompt entirely are others. Prompt caching only works when the stable spine remains sufficiently consistent; frequent changes reduce cache hit rates and erode the benefit. The choice depends on how stable the spine is and how the provider bills it. What matters is that content billed on every call becomes content billed once.
The assembled request touches every seat. Trace visibility of the outbound payload is one artefact; the responses differ.
Engineering. Log and inspect the full assembled request as it leaves the client. Add per-component token counts to observability so trims and additions to the prompt are visible in the same place as latency and cost.
Platform/Infrastructure. Expose payload composition — spine size, head size, tool-schema footprint — as first-class metrics alongside tokens and latency. Provide framework-level primitives for the spine/head split and for prompt caching where the provider supports it.
Architecture/CTO. Treat the assembled request as a system boundary, not a string. Decide where the stable spine lives, how it is versioned, and how changes to it are reviewed — the prompt is production code even when it looks like prose.
Sustainability/ESG. Every token in the assembled request incurs infrastructure, cost, and emissions overhead on every call. Report payload composition alongside call volume; a smaller, cached spine reduces per-call footprint without changing user-visible behaviour.
Business/Product. Product decisions determine what the agent must know and do — persona, capabilities, safety posture, tone. Each of those decisions writes tokens into the spine. Making those trade-offs explicit lets the team weigh the behaviour they are buying against the cost they are paying to ship it on every call.
Five seats, one payload: the request the model reads is a shared surface, shaped by decisions from every seat.
For the next agent or LLM feature you own, make the assembled request visible.
Start with the inventory. Capture one real outbound request end to end — everything the framework sends, in the order it sends it. Break it into components: system prompt, tool schemas, few-shot examples, formatting instructions, retrieved context, conversation history, user query. Count tokens for each. If any component is a surprise — content you did not know was shipping — that is the first finding.
Then trim. For each component, ask whether removing it changes the model’s output on a representative set of examples. Formatting rules that duplicate the model’s default behaviour, few-shot examples covering cases the model now handles reliably, and tone guidance the fine-tune already reflects are the usual candidates. Trim what does not earn its cost.
Then split. Draw a line between what changes every call (the head) and what changes rarely (the spine). Refactor the client so the two are assembled separately, even if they still ship together for now. The split is what makes the next rung possible.
Then amortise. Where the provider supports prompt caching, route the spine through it. Where the spine is truly stable, consider distillation into a fine-tune. Where a section of the spine is only needed on some routes, load it conditionally rather than shipping it on every call.
One caution: trimming and caching must be tested against real behaviour, not assumed. A component that looks decorative may be doing invisible work; a cache that fragments the spine may miss more than it hits. Measure the output, not the plan.
One inventory, one spine, one cache. That is the work.
If one thing you take away from today, let it be this:
The user writes the question. The framework writes the request. Only one of them is paid for by the user; both are paid for by you.
Day 18 stays with the system around the model and turns to another layer that sits quietly between the request and the response.