Rate Limit

Every online service runs on computers that can only handle a limited amount of work per second. To keep individual users from overloading these computers, providers set a fixed cap. A rate limit determines how many requests someone may make within a certain period of time. Typical rules look like “at most 60 requests per minute” or “at most 1000 per day”. Anyone who goes beyond that no longer gets a response, but an error message. Usually the block is only temporary: once the time window has elapsed, things carry on as normal.

What would happen without a cap

Computing power is a scarce and expensive resource. A single program firing off requests in an endless loop can bring a server to its knees for everyone else. The rate limit ensures that capacity is distributed fairly. So it is not a mechanism of harassment, but a protective measure for operations.

With AI services there is a second reason: cost. Every response from a language model consumes computing time on specialized graphics chips, and those are in short supply. Providers such as OpenAI, Anthropic, or Google therefore tier their limits by plan. Those who pay little may request little. Those who pay more get a higher limit. The rate limit is thus also a pricing instrument.

A third purpose is security. Anyone trying to guess a password tries out millions of combinations. A limit of five login attempts per minute makes this kind of attack practically impossible. The same principle slows down bots that want to scrape data from a website en masse.

The bucket that slowly refills

Technically, the server keeps count of how many requests come from a particular user. The user is usually identified by their access key, a kind of digital ID, or by their internet address. Each user has their own counter running. If the counter exceeds the allowed number, the server blocks the next requests.

A widespread method is called token bucket, that is, a bucket filled with coins. Every request costs one coin from the bucket. The bucket refills at a steady rate, for example one coin per second. If it’s empty, you have to wait. The advantage: short bursts with many requests are allowed, as long as you stay slow on average.

With AI interfaces, it is often not the number of requests that counts, but the amount of text. This is measured in tokens, that is, in word fragments. A long text consumes more of the quota than a short question. Frequently both limits apply at the same time: one for requests per minute, one for tokens per minute. Whoever breaks either one gets throttled.

Error 429 and what developers do about it

In everyday use, people usually only notice a rate limit once it kicks in. A chatbot suddenly responds that too many messages have been sent. An app displays the message “Too Many Requests” or the error code 429. Search engines and social networks also throttle when someone clicks or scrolls unusually fast.

Rate limits regularly show up in business news whenever an AI provider launches a new model. At first the limits are often very tight, because the necessary chips are lacking. Later they get loosened. Disputes between platforms and developers frequently revolve around this too: when X, formerly Twitter, drastically lowered its limits in 2023, many services built on top of it collapsed.

Programs that use an interface handle limits systematically. The so-called backoff approach is common: after a rejection, the program waits briefly, then a bit longer, then longer still. This way it finds on its own a pace the server can handle. Incidentally, a common misconception is that a rate limit is the same as a usage limit in a subscription plan. The rate limit restricts speed, not the total amount over the course of the month.