Day 20 — The Capacity That Waits
Day 19 covered the compute the team triggers itself — evals on merge, LLM-as-judge sweeps, notebook experiments — and walked the Meter, Gate, Sample, Substitute ladder that moves it from an unbudgeted reflex into a governed line on the bill. That closes the loop on compute triggered by people. There is one more category to name before this arc ends: the compute and the standing infrastructure that run when nothing is being asked at all.
Production AI systems rarely spin up on demand. The model endpoint holds a warm floor of replicas so latency stays inside contract. The vector database keeps its indexes loaded in memory so retrieval stays fast. The cache layer, the reranker, the orchestration runtime, the guardrail service, the observability pipeline — each of them stands up when the system launches and stays up while the system is production. All of these are patterns designed to protect latency, guarantee capacity, or meet compliance requirements. They also mean the whole supporting stack is on when traffic is not. Electricity draws. Cooling runs. Manufactured silicon sits at temperature, doing no work.
The per-call optimizations that filled the earlier issues in this series matter most when the bill scales with calls. Once capacity is standing, the lever shifts. Utilization becomes the number that matters, and the gap between provisioned capacity and used capacity becomes the gap on the bill — across the model fleet and across every piece of supporting infrastructure alongside it.
A retail platform team runs a customer-service assistant on a dedicated inference deployment, backed by a vector database for retrieval and a cache layer for warm responses. The stack was sized eighteen months ago against a projected peak traffic curve, with a safety multiplier layered on top. It has served production faithfully ever since. Nobody has re-examined the sizing on any component.
When a new sustainability programme asked for a utilization report across the full AI stack, the team pulled a month of metrics. The picture was consistent from top to bottom. Roughly two-thirds of the standing capacity across the stack had been warm and idle for the entire window.
The bill told the same story in three parts.
The autoscale floor. The model deployment autoscaled between a minimum and a maximum replica count. The maximum handled peak. The minimum, set high early on for latency safety, kept the fleet warm through overnight and weekend hours when real traffic dropped to near zero. Most of the wall-clock hours on the bill were minimum-floor hours, serving almost no requests.
The over-indexed retrieval store. The vector database was configured at its highest-precision defaults — full-dimensional embeddings across the entire corpus, an exact-search index built for peak query volumes well above what the feature actually saw. The retrieval workload would have returned the same top-k results from a smaller index, a lower-dimensional embedding, or an approximate search. The configuration had been chosen at setup and never revisited against the workload. The store scaled with dimensionality, index type, and corpus size — none of which had been sized to the queries it actually served.
The forgotten pilot. A dedicated endpoint and its supporting index traced back to a proof-of-concept from the previous year. The pilot had ended, the feature had shipped on a different stack, and neither the endpoint nor the index had been retired. Together they had been serving fewer than a hundred requests a month for over six months, on full-size deployments.
When the bill scaled with calls, per-call efficiency was the lever. Once the stack is standing, the bill runs on wall-clock — the hours capacity is on, not the hours it is used — and wall-clock does not care whether a request arrived.
The obvious objection is that standing capacity exists for good reasons. Cold starts are slow. Retrieval latency contracts are real. Compliance requirements can forbid multi-tenant serving. All of that is true. The argument here is not that standing capacity is wrong. It is that a stack sized for peak and never re-examined is a stack that will drift into low utilization across every component, and low utilization on standing capacity is one of the largest recoverable inefficiencies in production AI.
Four rungs move standing capacity from a fixed cost centre into a managed one. They apply to every provisioned component in the AI stack, not only to the model endpoint.
Size. Right-sizing is not only about how much capacity is provisioned; it is also about which capacity. For model endpoints, that means fitting replica count and instance type to observed workload. For vector databases and retrieval infrastructure, it means fitting index type, embedding dimensionality, and precision to the queries the feature actually serves — an exact-search index over full-dimensional embeddings is expensive when an approximate index over compressed embeddings would return the same top results. For caches and queues, it means fitting memory allocation and partition count to real hit rates and throughput. The starting point is measurement, not architecture. Pull utilization and workload data for every provisioned component over a representative window — a full month is a good default, since traffic patterns often have weekly and monthly rhythms — and compare average and peak against provisioned capacity. Right-sizing then becomes a defensible number rather than a guess.
Share. Many organisations run separate infrastructure for separate models, teams, or features by default. Modern serving frameworks support multi-model endpoints, where several models share GPU capacity. Vector databases support shared collections and shared indexes across features. Cache layers and observability pipelines can serve multiple workloads from the same fleet. Consolidating low-utilization components onto shared infrastructure often cuts total standing capacity significantly without changing anything user-facing. The trade-off is real — shared infrastructure adds coordination and isolation considerations — but for workloads that do not require dedicated capacity, sharing is usually the largest single lever.
Cycle. Standing capacity does not need to stand at the same size around the clock. Predictable low-traffic windows — overnight, weekends, holidays — are candidates for scheduled scale-down across model replicas, retrieval nodes, and cache capacity together. Non-production environments are candidates for scale-to-zero, with a modest cold-start penalty accepted for internal users. Autoscaling minimums and index replica counts that were set defensively at launch are candidates for re-tuning against actual observed troughs.
Retire. Stacks accumulate components the way filesystems accumulate files. Pilot endpoints, deprecated collections, orphaned indexes, unused caches, replaced architectures — the infrastructure often outlives the workload it was built for. Retirement needs to be a scheduled review, not an accident. A component with no owner, no dashboard, and no traffic is not free. It is a line on the bill and a share of manufactured silicon drawing power.
Standing capacity crosses every seat at the table, but it is often owned by none of them in aggregate.
Engineering. Every provisioned component gets an owner, a utilization target, and a review cadence. Provisioning becomes a decision with a follow-up date, not a one-time configuration.
Platform and Infrastructure. GPU, memory, and index capacity become first-class assets with utilization metrics. Capacity planning shifts from “how much do we need” to “how much of what we have is doing work.” The dashboards that already exist for CPU and memory extend to accelerators, to vector stores, and to every supporting layer.
Architecture and CTO. Stack size becomes a design parameter rather than a default. The question of dedicated versus shared serving, of exact versus approximate retrieval, of warm floor versus scale-to-zero, becomes an explicit architectural trade-off that is documented and revisited, not a decision made once at launch and inherited.
Sustainability and ESG. Idle capacity is a category of embodied and operational emissions that reporting has historically missed. The hardware was manufactured, shipped, installed, and powered whether or not it is doing work. Utilization data across the full stack brings that reality into the emissions boundary.
Business and Product. Cost per served request improves without changing the product surface. The margin argument and the sustainability argument point at the same operational discipline: infrastructure that is doing work is infrastructure that is earning its footprint.
Five seats, one stack.
The work begins with a utilization report and ends with a policy.
Export one month of utilization metrics across every provisioned component in the AI stack — model endpoints, vector databases, retrieval indexes, cache layers, orchestration runtimes, observability pipelines. For each component, record four numbers: provisioned capacity, peak observed utilization, average utilization, and the ratio of the two. Order the list by the largest gap between provisioned and used.
Pick the top component on the list. Assign it three things: an owner, a utilization target, and one operational change from the ladder — a resized floor, a shared deployment, a scheduled scale-down, a right-sized index, or a retirement. The target makes the gap visible. The owner makes the gap someone’s job. The change closes the gap.
Measure again next month.
One component, one owner, one policy. That is the work.
The request ends. The capacity does not.
Day 20 closes the arc on the system around the model. Days 21 to 25 open a different territory — the footprint that is already spent before a single call runs. The hardware in the rack, the manufacturing that produced it, the training that shaped the model on it. Emissions that do not scale with usage because they were paid in full before usage began.