Route before you spend

The arc is familiar. A team ships its first AI feature against the best model available, because that's the sensible way to find out whether the thing works at all. It works. More features follow, all pointed at the same endpoint, because that's the path of least resistance and the bill is still small. Nine months later the inference line has outgrown the compute line, someone asks what's driving it, and the answer is that nobody ever decided — the default was set during a prototype and then inherited by everything.

We've written before that we build the cost model before the code. Inference is the same discipline applied to a newer bill. Which model serves which call is an architectural decision, and like most architectural decisions it is cheap to make deliberately at the start and expensive to revisit once forty call sites have hardened around it.

Most calls in a working system are not hard calls

Look at the traffic in a mature AI feature and the requests sort themselves into tiers almost immediately.

There is a long tail of work that is mechanically simple and just happens to be expressed in language: classifying an inbound message, extracting fields from a form, deciding which of six routes a request belongs on, normalizing a messy string into a schema, drafting a sentence against a fixed template. These are the bulk of the volume in most systems we see, and they do not need a frontier model. They needed one in 2023. They don't now.

Then there is a much smaller set of calls that carry the actual product: the synthesis, the long-horizon reasoning, the judgment on an ambiguous case, the generation a user will read closely and notice the quality of. That work deserves the best model you can get and is usually worth paying for outright.

The mistake isn't spending on the second tier. It's paying second-tier prices for first-tier volume because the routing decision was never made.

Routing is a component, not a config flag

If tiering is an architectural decision, it needs to live somewhere in the architecture. In practice that means a seam between the caller and the model — one place that decides which tier a given request goes to, rather than a model name pasted into every call site.

A real seam buys you three things. You can change a tier assignment without touching feature code. You can measure spend and quality per tier instead of staring at one aggregate number. And you can move a class of work down a tier as the cheap models improve, which they do on a schedule that has nothing to do with your release cycle.

The decision of what can move down is not a guess, and this is where the eval harness earns its keep again. Routing without evals is a cost reduction you're hoping about. With a golden set for each class of work, moving a tier is a measurement: run the cheaper model against the cases you care about, look at where it breaks, and either accept the result or don't. The harness is what makes cost work safe to do — which is another way of saying that the teams who invested in evals early are the only ones who can economize later without flying blind.

The cheapest token is the one you never send

Routing is the largest lever, but it isn't the only one, and the others are mostly about discipline rather than cleverness.

Cache aggressively, at both ends — prompt prefixes that don't change, and results for requests that repeat. Be honest about how much context you're shipping on every call; a retrieval step that returns twenty chunks when three would do is a cost multiplier applied to your entire request volume. Look hard at retry and fan-out patterns, because a design that quietly issues five model calls to answer one question has a cost profile nobody reviewed. And know which of your calls are interactive and which can batch, because that difference is often priced.

None of this is exotic. It's the same posture as pricing a data-transfer pattern before it's welded into a diagram. The point is to look while looking is still cheap.

Re-measure, because the floor moves

Here's the part most teams miss, and it's the reason this note exists rather than a spreadsheet.

The capability floor — what the smallest, cheapest, most ordinary model can do reliably — has been rising fast, and most teams set their model defaults once and never checked again. The assumption that a task needs the frontier is frequently a snapshot of a judgment made eighteen months ago against a model generation that no longer represents the cheap tier. That assumption is now a line on an invoice.

So we treat the tier assignment as something with a review date on it, not a decision made once. Every few months, take the classes of work you route to the expensive tier, run the current cheap tier against the golden set, and find out whether the reason you're paying still exists. Sometimes it does. Often enough it doesn't, and you retire a cost you'd stopped seeing.

The conversation this lets you have

Modeling inference cost before you build has the same payoff as modeling cloud cost before you build: you get to tell someone what the system will cost to run while the answer can still be changed. Per-request cost, dominant driver named, and a defensible statement about what happens to the bill at ten times the volume — because the tiers were designed rather than discovered.

Decide what runs where, put the decision behind a seam, and hold it to a number. The system is cheaper, and you chose that on purpose.

---

Want your inference bill designed rather than discovered? Start a conversation.