Day 13: The Queue You Never Built

June 17, 2026 View as markdown

Most inference traffic is not waiting on a human. It is waiting on a loop. Embeddings for an index that refreshes hourly. Classifications for a moderation queue. Summaries for a digest that ships at seven in the morning. Evaluations that run overnight. None of these need a sub-second answer.

Yet almost all of them are dispatched as if they did. One request, one response, one connection, one forward pass. The producer code was written that way because that was the natural shape of a function call, and nothing has forced the question back open.

The principle for today is that latency tolerance is a design choice. When a team has not made it deliberately, the default is always urgency, and urgency is expensive.

A platform team pulls a month of inference logs to find the source of a cost spike. The spike is not in the production chat API. It is in the document embedding service. Every upload triggers its own embedding request, even though the index refreshes only every six hours. Nobody is waiting. The calls could have been pooled from the start.

The audit surfaces three patterns underneath this same shape, repeating across the platform.

The Solo Trickle. Background jobs that issue one request per item because the loop was written before batch endpoints existed, or because the team writing it did not know they did. The shape of the producer code locks in the shape of the bill.

The Synchronous Reflex. The unexamined assumption that every call must return inline. The majority of inference traffic in most platforms is not user-facing — it feeds indexes, populates dashboards, scores records overnight. Treating it like chat is the inheritance of a default that nobody chose.

The Latency Budget Nobody Set. Pipelines that have never been asked how long their answer can wait. When the question is finally posed to the owner, the answer is almost always longer than the system currently allows. The slack was there. The architecture was built without it.

The GPU does not care whether it processes one request or thirty in the same forward pass. Higher utilisation means more useful work per watt consumed. The same infrastructure delivers more output without proportionally increasing energy draw.

The objection arrives quickly. “We cannot make our users wait.” That is true for the slice of traffic that is genuinely user-facing and latency-bound. It is not true for most of what runs through inference. The audit almost always reveals that the largest share of compute is background work dressed in real-time clothing, because the real-time API was the easiest one to reach for. The work to be done is not to slow anything down. It is to stop pretending everything is urgent.

The move from urgent-by-default to deliberate-by-design follows a short ladder. Each rung is mechanical on its own. Together they reshape how inference flows through the platform.

Classify. Sort every inference workload by the latency it can actually tolerate, not the latency it currently delivers. Interactive belongs in one bucket. Near-real-time, where seconds are acceptable, belongs in another. Background, where minutes or hours are acceptable, belongs in a third. The third bucket is almost always the largest by volume and the most overlooked by design.

Queue. Place latency-tolerant work into a shared queue rather than dispatching it immediately. The queue is the structural change that everything else depends on. Without it, every later step is a workaround. With it, the producer side stops calling and starts enqueuing, and the consumer side stops waiting and starts draining.

Pool. Group queued requests that share a model, a parameter set, and a tolerance window. The pool, not the individual request, becomes the unit of dispatch. This is where the economics shift — many small calls collapse into one larger forward pass.

Window. Set an explicit holding window for each pool. Tens of milliseconds for latency-tolerant interactive. Several seconds for near-real-time. A full off-peak cycle for true background. The window is the lever that trades a small amount of delay for a large amount of efficiency.

Dispatch. Send the batch when the window closes or when it fills. Where the provider exposes a dedicated batch endpoint at a discounted rate, route to it. Where the inference engine supports continuous batching, configure it. Where neither is available, the queue itself still recovers most of the gain by raising the work done per forward pass.

The shape of this change looks different from each seat at the table, but every seat has a part in it.

Engineering. Find the loops that call an inference endpoint inside them. Replace the inner call with an enqueue. Build the drainer on the other side. The change is mechanical, and embedding and classification pipelines pay it back fastest.

Platform/Infrastructure. Make batch dispatch a first-class capability, not a workaround. Expose batch endpoints, configure continuous batching in self-hosted engines, and surface throughput-per-watt next to latency on every dashboard. Utilisation belongs on the sustainability scorecard.

Architecture/CTO. Treat batch-first as the default for non-interactive inference. Real-time becomes the exception that earns its premium. Everything else routes through the queue.

Sustainability/ESG. Track fleet utilisation as a leading indicator for Scope 2 emissions. Under-utilised accelerators consume power even when delivering little useful work — the idle draw is real, the output is not. Better batching increases work completed per unit of energy consumed, and grid draw moves with it.

Business/Product. Surface latency as a product choice rather than a system default. “Available in five minutes” is not a degradation. It is an offering. Most users do not need a nightly digest in real time, and most internal users do not need their analytics refresh to be synchronous.

Five seats, one principle: latency tolerance is a design choice, and the system bills you for the one you forgot to make.

The work begins with a single pipeline, not a platform-wide migration. The point is to make the pattern legible before scaling it.

Pick one background pipeline that currently dispatches inference calls one at a time. Ask the pipeline owner the only question that matters — how long can the answer wait. Refactor the producer side to enqueue, the consumer side to drain in batches. Where a batch endpoint exists for the model in use, route to it. Measure throughput-per-watt and cost-per-thousand-requests before and after. Publish the delta to the team that owns the pipeline.

Batching is not free of risk. A poorly sized window can introduce user-visible delay where none was budgeted for. A poorly sized batch can exhaust memory, trigger timeouts, or amplify a single failure into many. Start with the workload where latency is least sensitive and volume is highest. Tune the window upward until the throughput gain plateaus, then stop.

One pipeline, one window, one batch. That is the work.

The most expensive queue in AI is the one you never built.

The next issue closes the infrastructure chapter by asking where the work should run. Region, fleet, and time of day each carry a different carbon weight. Placement is the lever most teams have not yet pulled.