Rate Limit
Batas Laju
A cap on how fast you may call the API, and the most common cause of failures at scale.
A rate limit caps requests per minute, tokens per minute, or both, usually per organisation or per key. Providers set them to control load and to shape usage. Exceed one and the API returns an error, usually with a retry-after hint.
This is a normal operating condition, not an exception. Any production integration needs three things: a retry with backoff, a queue so bursts become a steady trickle, and a cap on your own concurrency so a traffic spike does not immediately exhaust the limit.
Rate limits also shape architecture. A limit of 60 requests per minute per key means a design where one worker does one request at a time needs a key per worker, and a design with 500 users needs a queue in front of the model, not a faster model.
Related terms
API
A defined way for one program to ask another for something.
Latency
How long you wait. The time to the first token matters more than the total.
Cost per Token
What you pay for each unit of text the model reads or writes.
Integration
Connecting your system to the other systems it needs to work with.
Model Provider
The company that serves a model to you. Usually someone else's server.
Want this working inside your team?
It starts with one conversation about your workflow.