Routing
Deciding which model or path handles a request before you send it.
Routing is choosing between paths before the work starts. A router sends easy requests to a small fast model and hard ones to a large capable one, and sometimes sends them to a human.
The payoff is cost and latency, and it is large: most production traffic is easy, and the expensive path is needed for a small minority of cases. A router that classifies the request as easy or hard is one of the highest-leverage components in an AI system.
The risk is misrouting. A hard request sent to the small model fails quietly and expensively, because nobody is watching. The fix is to make the hard case obvious, to keep a fallback path, and to measure the router itself on the cases where the two models disagree.
Related terms
Mixture of Experts
A model with many specialist sub-models, of which only a few run per token.
Cost per Token
What you pay for each unit of text the model reads or writes.
Latency
How long you wait. The time to the first token matters more than the total.
Model
The trained file of learned parameters that turns input into output.
Evaluation
Measuring whether the system does the job, on a fixed set of cases, repeatedly.
Want this working inside your team?
It starts with one conversation about your workflow.