--- title: Day 3 · Right-Size the Model type: newsletter date: 2026-05-22 source: linkedin summary: The smallest technology that solves the task is the right one — sometimes a frontier LLM, often a small one, sometimes a classical model, and sometimes no AI at all. Once you choose by task instead of by default, every cost on the bill moves with you. A… newsletter: Green AI Bytes draft: false --- The smallest technology that solves the task is the right one — sometimes a frontier LLM, often a small one, sometimes a classical model, and sometimes no AI at all. Once you choose by task instead of by default, every cost on the bill moves with you. A frontier model summarising a meeting transcript is a Ferrari delivering a pizza. It works. It also consumes an order of magnitude more energy, water, and cost than a small model that would have produced an output indistinguishable to the person reading it — the exact multiplier depending on the task, the family, and the provider. And sometimes, the right answer is not a smaller model. It is no model. > A regex extracting dates from text carries a tiny fraction of an LLM call's footprint — no GPU draw, no inference, no agent loop multiplier, no hallucination risk. The hardware floor is still there, as it is for every piece of software. But the bill above that floor collapses by orders of magnitude. Right-sizing is a ladder, not a switch. It runs from *"do you need AI at all"* to *"if you need AI, do you need a generative model"* to *"if you need a generative model, how large does it need to be."* Most teams skip the first two rungs and land on the third with a frontier model already chosen. Multiply that choice by every query, every user, every retry, every agent loop, and the gap between the smallest technology the task needed and the largest one the team shipped becomes the single largest line of waste in production AI today. It is also the easiest line to recover. Right-sizing is one decision, made once, at design time. It pays off on every query for the lifetime of the feature. Three forces quietly push every team to skip the ladder. The first is the **demo bias** — the model that impressed in the pitch is the model that ships. The second is the **fear premium** — a larger model feels safer, because no one ever gets asked *"why did you use the big one."* The third is the **default trap** — most AI platforms ship a flagship model as the default, and that default quietly becomes the default in every team's code. > The result, today, is a planet's worth of inference running on models the work did not need — and a meaningful share of it running on AI when no AI was needed at all. Right-sizing is not a single model swap. It is a four-step hierarchy that runs above the model layer and into the organisation: > **Avoid → Simplify → Right-size → Escalate.** This is the operating principle of efficient AI. Every other principle in this series sits somewhere on it. The most overlooked rung is the third one — not in the code, but in the platform. > The default model on your platform becomes the default model in every team's code. That dropdown becomes 80% of your organisation's inference. The platform's marketing decision quietly becomes an energy decision — multiplied across every feature, every customer, every query. What you size deliberately, you can route deliberately. What you route deliberately, you can shrink. And what you can avoid running on AI at all is the lightest line on the bill. **Engineering.** Walk the ladder in order. Before reaching for a model, ask whether a rule, a lookup, or a classical approach already solves the task. If a model is needed, start with the smallest credible one in the family, run your evals, and escalate only on measurable failure. The 80% of queries the small approach handles well never need to touch the large one again. **Platform / Infrastructure.** Your defaults are the lever. Offer a tiered menu — deterministic helpers, classical ML endpoints, small LLMs, large LLMs — with the smallest credible option as the default at each layer. One change to the default config, multiplied across every feature your platform serves. **Architecture / CTO.** Build the *Avoid → Simplify → Right-size → Escalate* hierarchy into the architecture itself. Non-AI paths, classical-ML paths, and small-LLM paths should be first-class citizens, not afterthoughts. A small-first router is the single most underused piece of infrastructure in production AI today. **Sustainability / ESG.** Technology mix is the metric. Measure the share of features running on no-AI, classical ML, small LLMs, and large LLMs across the organisation, and report it monthly. It is one of the few sustainability numbers that moves *down* without anyone having to do less. **Business.** Stop specifying model names in product requirements. Specify quality bars and let engineering choose the smallest technology that hits them. Routing by task is more honest than routing by tier — and costs an order of magnitude less. Five seats, one ladder, one bill that moves. One thing to do this week: pick one AI feature in production. Walk it down the ladder. > *Does this task need AI at all, or would a deterministic approach solve it?* *If it needs AI, does it need a generative model, or would a classical one do?* *If it needs a generative model, is it running on the smallest one that holds quality?* Run the test for whichever rung your feature is sitting on. If a smaller technology holds quality on your evaluation set — ship the swap. Write down the energy, cost, and latency difference before and after. Share those three numbers with the five seats from The Lens. That single swap, on a single feature, is what the next 27 issues build on. > The smallest technology that solves the task is the right one. Sometimes that is a frontier model. Often it is a small one. Sometimes it is no model at all. The smartest system is rarely the largest — it is the one that knew when less was enough. --- Coming up next. Right-sizing chooses the technology once. But every call to that technology carries a second decision — what you send to it, and what you do with what comes back. The largest source of recoverable waste after right-sizing is the work the model never needed to do in the first place. That is the next principle. See you in the next issue. ---