Caching
Cache
Storing results so the same work is not repeated, and the cost drops with it.
Caching means storing a result and reusing it when the same input comes back. There are three levels worth knowing.
Application cache: you keep the answer and skip the model call entirely. Best when questions repeat, which they often do inside a company. Semantic cache: you embed the new question, look for a close previous question, and reuse that answer if it is close enough. Prompt cache: the provider notices that the start of your context is identical to a previous request and charges less for those tokens.
All three share the risk of staleness. A cached answer is only correct while the underlying data is unchanged, and a semantic cache is only correct while the similarity threshold is high enough that near-identical questions really do have near-identical answers.
Related terms
Cost per Token
What you pay for each unit of text the model reads or writes.
Token
The unit a model reads and writes. Roughly word-sized, but not exactly.
Embedding
A list of numbers that places text near other text with similar meaning.
Latency
How long you wait. The time to the first token matters more than the total.
Model Provider
The company that serves a model to you. Usually someone else's server.
Want this working inside your team?
It starts with one conversation about your workflow.