Temperature
The dial for randomness. Low is repeatable, high is surprising.
Temperature rescales the probabilities before sampling the next token. Near zero, the model takes the most likely token almost every time, so the same input gives the same output. Higher, flatter probabilities get a chance to win, so the output varies and can be more creative.
For extraction, classification, and code generation, low temperature is usually right. For brainstorming copy, high temperature explores more. A common mistake is raising temperature to make answers "better" when the real problem was an ambiguous prompt, which temperature can only make worse.
Related terms
Hallucination
Output that is fluent and wrong. Two different problems hide under one word.
Evaluation
Measuring whether the system does the job, on a fixed set of cases, repeatedly.
LLM
A model trained on a huge amount of text to predict the next token. The engine behind most AI products.
Non-determinism
The same input can produce a different output each time. Normal, and worth designing around.
Want this working inside your team?
It starts with one conversation about your workflow.