Rate Limit

A rate limit is an upper bound on the number of requests within a certain time. Anyone who uses an online service is constantly sending such requests: every search, every click, every question to a chatbot is one. The operator sets how many of these are allowed per minute, hour, or day. If the limit is exceeded, the service rejects further requests, usually with an error message. The message often reads something like: too many requests, please try again later. You can picture it like a traffic light at a highway on-ramp that only lets one car through every few seconds.

Why providers need an emergency brake

Servers have limited capacity. If too many requests arrive at once, the service becomes slow for everyone or fails completely. A rate limit ensures that a single user cannot claim the entire performance for themselves. So it is less of a nuisance and more of a distribution rule.

There is also protection against abuse. Anyone trying to guess a password does not try three variants, but millions. A limit of five failed attempts per minute makes such attacks practically impossible. Similarly, this prevents automated programs from scraping entire databases.

With AI services, there is a third reason: money. Every response from a large language model costs the provider computing time on expensive specialized chips. Without a limit, a single customer could rack up costs in the millions within a few hours. Rate limits are therefore also a business management tool.

Buckets, windows, and waiting times

Technically, the service keeps count of how many requests arrive from a particular source. This source is usually identified by a key, i.e. a kind of personal password for programs, or by the device’s network address. Each key has an associated counter that is reset at regular intervals.

A common method is called Token Bucket. Picture a bucket into which tokens continuously drip. Each request consumes one token. If the bucket is empty, you have to wait. The advantage: anyone who has been inactive for a while has a full bucket and is allowed to make many requests at once for a short time.

AI interfaces usually have two limits at the same time. One counts requests per minute, the other counts so-called tokens, i.e. the text chunks that make up the input and output. A single very long request can thus blow through the limit even though it was only one request. A common misconception is that a rate limit is the same as a price cap. In fact, it only limits the pace, not the total amount over the month.

From chatbot to trading app

In everyday life, people usually only notice rate limits once they kick in. In free versions of chatbots, things stop after a certain number of messages, followed by a pause or a weaker model. Anyone who repeatedly logs into an account with the wrong credentials gets locked out for a few minutes. Social networks also throttle activity when someone scrolls or posts unusually fast.

For developers, the topic is even more important. Anyone building an app that uses a provider’s AI model is assigned fixed limits. These often increase with payment history: new customers start low, long-standing customers are allowed more. Well-built programs catch a rejection and try again after a short wait.

Rate limits make the news when providers tighten or loosen them. When a major social media platform suddenly limited the daily number of readable posts in 2023, it was a topic of discussion for weeks. Rate limits also play a role with financial data: stock exchange interfaces often allow only a few queries per second, which directly affects fast trading programs.