Prefix Cache
The provider-side store that lets consecutive requests skip re-processing a shared prefix.
The prefix cache is the provider-side store that lets consecutive requests skip re-processing a shared prefix. When the start of a request matches the start of a recent one, same system prompt, same history up to some point, the provider reuses its prior work and bills those tokens at the cache rate.
The cache pays off because sessions grow append-only. Every request re-sends the whole history as input, and in a normal session the history only changes at the end, each request is the previous one plus a few new messages. The provider processes the long shared beginning once, stores the result, and picks up where the prefix ends.
Caches expire. How long an entry stays warm varies per provider, typically minutes rather than hours. Leave a session idle past the window and the next request rebuilds the prefix at full price once before caching resumes. The visible effect is that requests after a long pause cost more than the ones before them. A harness that injects the current time into the system prompt every turn breaks the prefix at the first changed token, and every request after that is billed at full rate.
Related terms
Cache Tokens
Input tokens the provider cached from a previous request, billed at a much lower rate.
Input Tokens
Tokens the harness sends on each request. Billed at a lower rate than output tokens.
Model Provider Request
One round-trip from the harness to the model provider. The unit that tokens are billed on.
Caching
Storing results so the same work is not repeated, and the cost drops with it.
System Prompt
The standing instructions a harness sends on every request.
Want this working inside your team?
It starts with one conversation about your workflow.