Token
The unit a model reads and writes. Roughly word-sized, but not exactly.
Text reaches a model as tokens, produced by a tokenizer: a fixed vocabulary of tens of thousands of fragments learned before training. Common words are often one token; rare or long ones split into several. The model never sees characters or words, only token IDs.
Everything is priced and measured in tokens. A token is roughly three-quarters of an English word, so a thousand tokens is about 750 words. Code is less predictable: a hash such as `a3f9c2e1` never appeared in the tokenizer's training material, so it splits into many tokens, while `function` is one. That is why a small file full of unusual strings can eat a surprising share of the context window.
Related terms
Context Window
Everything the model sees on one request. Finite, and it decides what fits.
Cost per Token
What you pay for each unit of text the model reads or writes.
LLM
A model trained on a huge amount of text to predict the next token. The engine behind most AI products.
Prompt
The input you give a model. The word covers more than the question you type.
Want this working inside your team?
It starts with one conversation about your workflow.