Day 15 — The Route Before the Model

June 26, 2026 View as markdown

The model is usually the most expensive component in an AI system. The compute it consumes, the memory it occupies, and the energy it draws all converge on a single moment: inference. Yet in many production systems, every request still reaches the model.

That is not architecture. That is reflex.

Token spend, caching, batching, placement — these are levers inside and around the call, and they matter. But there is a lever one rung higher, and in many systems it is the largest of them all: the route a request travels before the model is ever invoked.

A telecom company launched an LLM-powered support assistant. The press release celebrated end-to-end generative coverage. Three months later, an internal review pulled the request logs apart by intent. The pattern was uncomfortable: a large share of traffic was account-status lookups — “what’s my balance,” “when is my bill due” — requests with a deterministic answer sitting one database query away. Another large share was password resets and plan changes, flows that had worked for a decade without any model in the loop. A small remainder was the actual generative work the assistant had been built for. The model was answering every request because the system had no route that ended anywhere else.

Three patterns recur across teams that retrofit routing onto an already-shipped LLM product.

The Reflex Call. Every input is dispatched to the model. There is no pre-check, no classification step, no fallback path. The pipeline has a single shape, and every request follows it to the flagship model.

The Hidden Lookup. Requests with deterministic answers — account balances, order status, business hours, FAQ entries — are routed to the model because the assistant is the only interface the user sees. The model paraphrases what a database call could have returned in milliseconds.

The Default Heavyweight. The flagship model is the default for every request. Smaller models exist in the catalog, but they are reserved for experiments. Production traffic flows to the largest tier regardless of complexity.

Teams that run this exercise are often surprised by what they find. A meaningful share of production traffic — anywhere from a third to two-thirds, in our experience — turns out to be deterministic lookups, transactional flows, or simple retrieval tasks that never needed a flagship model in the first place.

The cheapest token is the one that is never generated.

A reasonable objection arrives quickly. Classification and routing are not free. They add latency, complexity, and their own compute cost. But a lightweight classifier in front of a large model is usually far cheaper than invoking the large model unnecessarily — a result demonstrated by cost-aware cascading research such as FrugalGPT (Chen et al., 2023).

The work resolves into four rungs, each one a decision the system makes before the next becomes necessary. Together they turn a single funnel into a graded path.

Filter. Reject what should not enter the pipeline at all. Malformed inputs, duplicates already in flight, requests from abusive sources, queries outside the system’s scope — these never need to see the model. A simple set of rules and rate checks at the edge removes a measurable share of traffic before any inference begins.

Classify. Determine the shape and complexity of what remains. Is this a factual lookup, a transactional flow, a generative task, or a multi-step reasoning problem? A lightweight classifier — sometimes a small language model, sometimes a fine-tuned encoder, sometimes a few rules — answers this question fast and cheaply. The classification becomes the routing key.

Route. Send each request to the smallest competent system. A database lookup beats a small model. A small model beats a large one. A large model is invoked only when nothing smaller will do. Model providers now make this explicit, with tiered model families designed for exactly this pattern.

Call. Invoke the large model when, and only when, the request has earned it. The flagship is now a destination, not a default. The traffic that arrives there is the traffic that genuinely requires it, and the carbon spent on those calls is carbon spent well.

The goal is not to minimise model calls in absolute terms. The goal is to ensure every model call is one that a cheaper route could not have answered.

The route before the model is a shared piece of infrastructure, and the pattern reads differently from each seat at the table.

Engineering sees a routing layer to build and maintain — a classifier to train, a fallback to wire up, a confidence threshold to tune. The work is real, but it is bounded, and the operational savings tend to dwarf it.

Platform and Infrastructure sees request volume to the largest tier fall, not because product usage fell, but because the right requests are now reaching the right tier. Capacity plans loosen. Quota pressure eases.

Architecture and CTO sees a system whose components are sized to their job. The pipeline is no longer a single funnel ending at the flagship; it is a graded path with multiple exits, and the design admits where deterministic logic still wins.

Sustainability and ESG sees the largest single contributor to inference carbon — flagship-tier traffic — coming down without a corresponding drop in service quality. The avoided emissions are real, attributable, and defensible.

Business and Product sees latency improve for the bulk of requests, because lookups and small-model responses arrive faster than flagship responses ever did. User experience improves alongside cost.

Five seats, one route.

The work begins with measurement, not architecture. Routing only earns its complexity when the traffic justifies it, and the traffic only reveals itself when it is read honestly.

Pull one week of production traffic. Label each request by intent, and by the minimum competent tier that could have answered it: rule, lookup, small model, or flagship model. The labels do not need to be perfect; they need to be honest. The exercise almost always surfaces that a meaningful share of traffic is being answered by a model when something smaller would suffice.

From there, choose one intent class to gate first. Pick the largest one — usually a class of deterministic lookups — and build the route that handles it without the flagship. Wire in a fallback so that low-confidence classifications still reach the model. Measure the share of traffic redirected, the latency change, and the avoided compute.

The caution is real. A poorly tuned classifier can route requests to a path that cannot answer them, and the user pays for that in a wrong answer or a frustrating loop. The mitigation is conservative thresholds, transparent fallback, and continuous monitoring of misroutes. Start with the intents where confidence is highest and the fallback is cheapest.

One classifier, one route, one fallback. That is the work.

The work of routing distils to a single line.

The model is the destination. Make it the last one.

Day 16 turns from routing to grounding. Once the call is justified, the next question is what enters the model — and the shape of the context window is the next carbon lever in the system around the model.

Chen, L., Zaharia, M., & Zou, J. (2023). FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. arXiv:2305.05176. https://arxiv.org/abs/2305.05176