Rate Cap

A rate cap is a fixed upper limit on the use of an online service within a certain time period. A provider might set, for example: at most 60 requests per minute, or at most 50 image generations per day. Anyone who reaches this limit no longer gets a response, but an error message instead. Usually you just have to wait until the time period has elapsed and the counter resets to zero. The English term literally means a “cap” placed on the usage rate. The term rate limit is also used for this, and in practice both mean the same thing.

Why providers cap usage

AI services cost real money with every single use. Behind each response is computing time on specialized chips that are scarce and expensive. Unlike a normal website, a request is therefore not nearly free. Without an upper limit, a single user with an automated script could rack up costs in the thousands within a few hours.

The second reason is fairness among users. A provider’s computing capacity is limited at any given moment. If a few customers were to occupy it entirely, response times for everyone else would become unusably long. A rate cap distributes the scarce capacity instead of allocating it on a “whoever clicks fastest” basis.

Third, rate caps protect against attacks and abuse. They make it harder to mass-test passwords or systematically probe a model through millions of requests. They also serve as a sales tool on the side: the free version of a chatbot has tight limits, the paid subscription more generous ones. This is exactly where most people first notice that rate caps exist at all.

How the counter works in the background

Technically, the provider keeps a counter for each user. With every request, it increases by one. Once the limit is reached, the server rejects further requests and sends back an error code, usually number 429, meaning “too many requests.” The response often also states how many seconds to wait.

There are two common methods for the time period. With a fixed window, the counter resets at the start of every full minute. With a sliding window, the service always counts the last 60 seconds from now. A popular image for the second variant is a bucket with holes: every request pours in a drop, while something is constantly draining out the bottom. If the bucket overflows, that’s it.

For language models, often not just the number of requests is counted, but also the amount of text. This is measured in tokens, i.e., word fragments a few letters long. A typical limit might then be: 500 requests and 200,000 tokens per minute. A long text can thus blow through the limit even though it was only a single request.

Rate caps in everyday life and in the news

Rate caps are most visible in chatbots. Sentences like “You’ve reached the limit for this model, try again in three hours” are nothing other than a rate cap in action. Image generators, translation services, and coding assistants also work with such limits. Anyone who has ever been annoyed that a tool suddenly stopped working has usually simply exhausted the limit.

In business news, rate caps show up as a signal of scarce computing capacity. If a provider lowers limits shortly after launching a new model, that often points to overloaded data centers. If it loosens them, new chips are in operation or the model is running more efficiently. For industry observers, limits are therefore a small leading indicator.

A common misconception is that a rate cap is the same as a cost cap. That’s not quite right. A rate cap limits the pace of usage, while a spending limit caps the bill at the end of the month. Many providers combine both, because otherwise a faulty program would generate costs slowly but relentlessly.