Inference
Inferensi
Running a trained model to produce an answer. What happens on every request.
Inference is the other half of the life cycle. The model is loaded, the prompt is turned into tokens, and the model generates one token at a time until it decides to stop. Every chat message, every autocomplete, every API call costs inference, and inference is the part you pay for per use. It is also the part you can optimise: smaller models, caching, and shorter contexts all reduce it.
Related terms
Model
The trained file of learned parameters that turns input into output.
Training
The expensive process that sets a model's parameters by exposing it to data.
Cost per Token
What you pay for each unit of text the model reads or writes.
Latency
How long you wait. The time to the first token matters more than the total.
Want this working inside your team?
It starts with one conversation about your workflow.