Latency
Latensi
How long you wait. The time to the first token matters more than the total.
Latency is the delay between sending a request and getting output back. It has two very different parts. Time to first token is how long until the model starts writing; total time is until it finishes.
For a chat interface, time to first token is what users feel. Ten seconds of waiting for the first word feels like a hang even if the whole answer takes twelve seconds. For a batch job, only total time matters.
Streaming exists to fix exactly this. Sending tokens as they are produced turns a silent wait into a visible progress bar, and it usually makes people perceive the system as faster even when it is not. The real wins come from smaller models, fewer round trips, and removing serial tool calls that could run in parallel.
Related terms
API
A defined way for one program to ask another for something.
Cost per Token
What you pay for each unit of text the model reads or writes.
Model Provider
The company that serves a model to you. Usually someone else's server.
Agent Loop
The repeated cycle of decide, act, observe. Where autonomy actually comes from.
Streaming
Sending the output as it is generated instead of waiting for the whole answer.
Want this working inside your team?
It starts with one conversation about your workflow.