Cheap Tokens, Uneven Answers: Auditing the Quality Cost of Dynamic Model Routing
Photo: Columbus Metropolitan Library, No restrictions, via Wikimedia Commons
Most NLP platforms operating in the United States today no longer send every request to a single model. A request arrives, a lightweight classifier or rule set scores it, and the system determines whether a small, inexpensive model can handle the work or whether the request should be escalated to a frontier model. Finance teams appreciate the arithmetic. Platform engineers appreciate the latency profile. What receives far less scrutiny is the third variable in that equation: the quality that quietly leaks out at the seams between tiers.
Call it the router tax. It is the aggregate quality cost of letting a decision function, rather than a fixed policy, choose which model answers each request. The tax is not a reason to abandon routing. It is a reason to measure routing as a system rather than as a cost lever.
How a Router Actually Decides
Production routers generally fall into four families, and each fails differently.
Rule-based routers inspect length, language, keyword presence, or request origin. They are transparent and cheap, but they encode assumptions that age quickly. A threshold tuned on support tickets from a prior quarter may not hold once marketing campaigns shift the vocabulary of incoming traffic.
Classifier-based routers train a small model on labeled examples of requests that small models handled well. The router is now a model with its own training distribution, its own drift, and its own blind spots.
Cascades run the small model first and escalate when a confidence signal falls below a threshold. This approach depends entirely on whether that confidence score is calibrated — and confidence scores from different model families are not comparable quantities, even when they share a name.
Learned policies, including bandit-style optimizers, adjust routing based on observed outcomes and cost. They are the most adaptive and the least interpretable, and they will happily discover shortcuts that optimize the reward function rather than the user experience.
Where the Decision Logic Degrades
The failure modes cluster in a few predictable places.
First, the router drifts independently of the models it routes between. Traffic composition changes — a new customer segment, a seasonal surge, an integration that floods the queue with short transactional text — and the router's notion of a hard request goes stale.
Second, thresholds are usually set once, during a pilot, using data from a favorable period. Nobody revisits them after the small model is fine-tuned or the frontier model is upgraded.
Third, escalation paths can loop. A failed parse triggers a retry at the same tier, the retry fails again, and the request either times out or escalates late enough that the latency savings evaporate.
Fourth, and most insidiously, the router learns from proxy features rather than meaning. A prompt containing a long policy citation may be routed to the large model not because it is difficult but because it is long — while a short, ambiguous request with real ambiguity stays cheap and wrong.
Variance Hides Behind Averages
Aggregate accuracy is the wrong dashboard for a routed system. Consider a plausible configuration: 78 percent of traffic goes to a small model that answers 91 percent of its assigned requests correctly; 22 percent escalates to a frontier model answering 96 percent correctly. The blended figure is roughly 92 percent — comfortably close to the frontier number in a slide deck, and a poor description of reality.
What matters is the conditional distribution. If the small model's errors concentrate in a specific intent, language, or document type, then a subgroup of users is experiencing a materially worse product while the blended metric reports steady health. Segment accuracy by routing decision, by intent, by locale, and by input length. Plot the tail, not the mean.
The Evaluation Blind Spots
Offline evaluation sets are usually assembled from historical traffic and graded by a single reference model. That creates two problems for routed systems.
- The mix is wrong. Your eval set may reflect a traffic distribution that no longer exists, or one in which the router would have made different decisions.
- The grader is biased toward one tier. If the judge model resembles the frontier tier, small-model outputs that are correct but stylistically different will be penalized, and the router will look worse than it is — or the reverse, if the grader is lenient.
The practical fix is conditional evaluation: for every request in the suite, record what the router would have decided, then evaluate small-model and large-model outputs on the same inputs. This matched-sample design produces the one number that matters — the quality delta between tiers on the exact requests where the decision is being made.
Putting a Number on the Tax
Four metrics make routing accountable without requiring a research team.
- Escalation regret: the share of escalated requests where the small model's answer would have been equally acceptable. High regret means you are paying frontier prices for easy work.
- Missed escalation: the share of requests kept at the small tier that fail human or automated review. This is the tax that reaches users.
- Latency-adjusted quality: quality per request weighted by end-to-end latency, including retries, because a cheap answer delivered after three failed attempts is not cheap.
- Decision stability: how often the same input, replayed later, receives a different routing decision. Instability is a leading indicator of drift.
Log the router's decision, its score, and the features that produced it on every request. Without that record, post-incident analysis becomes guesswork.
Operating Routing as a Product
Treat the router as a component with its own owners, tests, and release cadence. Maintain a regression suite that specifically targets the decision boundary — inputs engineered to sit just above and just below the escalation threshold. Run quarterly shadow evaluations in which a sample of traffic is answered by both tiers and compared. Keep a documented escape hatch that forces all traffic to a single tier during an incident, and rehearse using it.
Dynamic routing is sound engineering. But a router that is never audited is simply an unmeasured discount on quality — and in production NLP, discounts of that kind are usually paid by the user.