Day 19 — The Test Tail

July 10, 2026 View as markdown

Day 18 covered the data that AI systems leave behind — vector stores, indexes, embeddings that outlive the features that created them. That was the persistence side of the invisible bill. There is a compute side to it as well, and it fires before a feature ever reaches a user. Every benchmark run, every evaluation sweep triggered by a merge, every notebook experiment, every pass of one model grading another over a dataset is inference. It consumes the same inference infrastructure, incurs the same model costs (or equivalent infrastructure costs), and draws energy from the same grid.

This compute rarely appears in sustainability dashboards. It is paid out of a different budget line — engineering or R&D — and reported, if they are reported at all, under “infrastructure” or “experimentation.” When teams report on AI carbon, they usually report on what customers triggered. The compute they triggered themselves, in the process of building for those customers, sits in a separate column that nobody adds up.

The point is not to run fewer evals. It is to recognise that development-time inference is inference, and to apply to it the same discipline that has already been applied to the production path.

A machine learning platform team at a mid-size SaaS company completed a six-month effort to bring production inference costs down. Prompt caching, right-sized routing, output shaping — the full playbook from earlier in this arc. Production inference became dramatically cheaper. The monthly AI bill barely moved.

The gap, when they pulled the bill apart, was the development pipeline.

Eval-on-merge. Every pull request into the model-serving repository triggered a full evaluation sweep — one thousand samples across three benchmarks, run against a frontier model. The policy had been added early on for safety and never revisited. Roughly two-thirds of merged pull requests changed documentation, infrastructure, UI code, or unrelated services. Those PRs paid for the sweep anyway.

The judge that scaled. A small model handled routing decisions in production. In evaluation, a larger model judged the small model’s outputs — a pattern usually called LLM-as-judge, where one model scores another’s answers. Every evaluation sample required at least two model calls: one to generate the answer and another to judge it. Some suites required a second judge to resolve disagreements. The most expensive calls in the eval pipeline were the ones doing the grading, not the ones being graded.

The prompt sweep. Prompt engineering ran as an ad hoc sweep — a researcher would test twenty prompt variants across a five-hundred-sample validation set, log the results, and move on. The runs were untracked and unbudgeted. Most variants were discarded within an hour of the sweep completing. The compute remained on the bill.

The product ships once. The evals run every day. Both are on the bill.

The obvious objection: evals are how AI systems ship safely, and cutting them means shipping worse. That is true, and it is not what this issue argues. Eval discipline is not about testing less. It is about testing deliberately. A sweep that fires on every PR is not more rigorous than one that fires on the PRs that changed the model — it is more expensive at the same rigor.

Four rungs move development-time inference from an unbudgeted overhead into an observable, governed part of the AI system.

Meter. Development inference must appear in the same dashboards as production inference. That means labelling every provider call — by request header, by API key, by project, or by account boundary — so that eval, notebook, sweep, and batch traffic are separable from user-triggered traffic. Until dev inference is metered, every downstream decision about it is a guess.

Gate. Not every trigger deserves a full sweep. A change to a Kubernetes manifest does not need to run three benchmarks against a frontier model. Eval pipelines belong behind a router of their own — a small classifier or a set of path-based rules that decides which changes warrant which suites. The default is the smallest useful eval, with escalation reserved for changes that actually touch model behaviour.

Sample. Full-dataset sweeps are appropriate for release gates. They are rarely appropriate for iterative development. Most eval runs during a workday can rely on representative subsets, chosen once and reused, with the full sweep reserved for the moment before a change ships. A subset one-tenth the size often reduces runtime and cost by nearly the same proportion while preserving rapid feedback.

Substitute. The judge does not always need to be the largest model. For many scoring tasks — classification agreement, rubric adherence, structural checks — a smaller judge is competitive, and the one-time meta-evaluation to prove it earns back quickly. Reserve the frontier model for the judgements that genuinely require it: nuanced correctness, ambiguous rubrics, cases the smaller judge cannot separate.

Development inference touches every seat at the table, but it is claimed by none by default.

Engineering. The eval config becomes a decision surface. Which triggers fire which suites, on which datasets, with which judge — each answered explicitly and versioned alongside the code it evaluates.

Platform and Infrastructure. Dev-time inference joins production inference in the same observability plane. Capacity planning becomes honest — the total AI footprint is the union of what customers ask and what the team asks to check them.

Architecture and CTO. Development compute is recognised as a first-class part of the AI system’s cost and carbon profile, not an unbudgeted overhead absorbed by R&D. The architecture diagram grows a second lane — the path a change travels before it reaches the customer path.

Sustainability and ESG. The reporting boundary widens. Emissions accounting for AI includes the inference the team itself generated, not only the inference customers generated. The number goes up in the first quarter it is measured; that is the point.

Business and Product. Faster iteration and lower spend are not in tension. A team that samples and gates its evals ships changes sooner and pays less to prove they are safe. The velocity argument and the sustainability argument converge on the same practice.

Five seats, one tail.

The work begins with a bill and ends with a policy.

Export one month of AI provider usage and billing data. Split it into two columns: production and non-production. Non-production covers evals, notebooks, sweeps, batch scoring, LLM-as-judge runs, and anything else triggered by the team rather than by a customer. Group each line item in the non-production column by workload:

CI
notebooks
evals
prompt sweeps
batch jobs
production

The exercise almost always surfaces at least one workload that is spending more than anyone had realised.

Pick the largest single trigger. Add two things: a budget cap and a sampling policy. The cap makes the trigger visible when it exceeds itself. The policy defines the default eval size and the escalation rule for full sweeps. Together, they move that trigger from a reflex into a decision.

Measure again next month.

One bill, one trigger, one policy. That is the work.

The product ships once. The evals run every day.

Day 20 closes the arc on the system around the model. From the compute the team runs testing, to the compute that runs when nothing is being asked at all — the standing capacity of production AI, and what it costs to keep the lights on between calls.

ISO/IEC 21031:2024 — Software Carbon Intensity (SCI) specification.
Green Software Foundation. Software Carbon Intensity — measurement guidance. greensoftware.foundation
Zheng, L. et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv:2306.05685
Anthropic. Prompt caching — pricing and eligibility. docs.anthropic.com
OpenAI. Evals — framework and best practices. github.com/openai/evals