Mixture of Experts
A model with many specialist sub-models, of which only a few run per token.
A mixture of experts model keeps many parallel sub-models, called experts, and a router decides which experts handle each token. Because only a few experts activate per token, the model can have the total capacity of a very large model while paying only for the part it uses.
This is a common trick for cutting serving cost without cutting quality. The catch: all experts still have to live in memory, so the model file is large even though the compute per token is small. It works best when you have the hardware to hold it and serve many requests.
Related terms
Model
The trained file of learned parameters that turns input into output.
Quantization
Storing weights in fewer bits so a model runs with less memory and cost.
Cost per Token
What you pay for each unit of text the model reads or writes.
Routing
Deciding which model or path handles a request before you send it.
Want this working inside your team?
It starts with one conversation about your workflow.