Streaming
Sending the output as it is generated instead of waiting for the whole answer.
Streaming is a transport pattern where the response arrives in pieces as the model generates them, usually over server-sent events. The user sees the answer being written.
It matters for perceived speed more than actual speed, and it matters more for long answers than short ones. It also changes failure handling: with a stream, you can show partial output and know the request succeeded as soon as the first chunk arrives.
The cost is complexity. You need to handle a connection that can drop mid-answer, and decide what the user sees when it does. Many applications end up buffering the stream anyway to validate it, which throws away the benefit.
Related terms
Latency
How long you wait. The time to the first token matters more than the total.
API
A defined way for one program to ask another for something.
LLM
A model trained on a huge amount of text to predict the next token. The engine behind most AI products.
Cost per Token
What you pay for each unit of text the model reads or writes.
Want this working inside your team?
It starts with one conversation about your workflow.