Throttling

Throttling means that a system deliberately slows down or rejects requests once a certain limit has been reached. This limit is usually the number of requests per minute or per day. If a user exceeds it, they receive an error message or have to wait until the quota renews. The term comes from English and literally means “to choke” or “restrict” — that is, reducing speed or flow. In technology, this principle applies wherever a limited resource — for example computing power, storage, or network capacity — has to be shared among many users.

Throttling as a protection and control tool

Without throttling, a single user or a single program could block a system for everyone else. Imagine a hundred users sharing one server. If one of them sends a thousand requests per second, there’s hardly any capacity left for the remaining 99. Throttling prevents exactly that by assigning each user a fixed quota.

At the same time, throttling is a business model. Many providers offer free access with low limits and sell higher quotas in paid tiers. Anyone using the free version of the OpenAI API is allowed significantly fewer requests per minute than a paying enterprise customer. The limit doesn’t force users directly into a purchase — it simply makes the upgrade more attractive.

How quotas are measured and enforced

The most common method is called the Token Bucket. Each user has an imaginary bucket that fills up at regular intervals with markers — so-called tokens. Each request costs a certain number of these tokens. If the bucket is empty, new requests are rejected until it refills. This allows short load spikes while preventing sustained abuse.

There is also the simpler sliding window method: the system simply counts how many requests have come in over the last 60 seconds. If a threshold is exceeded, a block follows. Both methods run fully automatically, usually on servers positioned upstream of the actual service — the actual content isn’t even accessed if the limit has already been reached.

It also matters exactly what is being measured. Some systems limit the number of requests, others the data throughput in megabytes, and AI services like GPT often count so-called tokens — the smallest language building blocks that make up text. A long request then costs more than a short one, even if both are technically just a single request.

Throttling in AI products and everyday life

Anyone who uses ChatGPT heavily in the free version will sooner or later run into a message like “You’ve reached your limit.” That’s throttling. Music services like Spotify also occasionally throttle download speed for offline content if a user downloads an unusually large number of tracks in a short time. Mobile carriers speak of throttling when they reduce internet speed to just a few kilobits per second after the monthly data quota has been used up.

In reporting on AI companies, the term mainly comes up in connection with APIs — that is, interfaces through which developers integrate AI models into their own applications. When a provider like Anthropic or Google expands its capacity, it often simultaneously announces higher throttling limits. If demand grows faster than infrastructure, limits are tightened instead. Throttling is thus an indirect indicator of a service’s operating state: tight limits signal that demand currently exceeds supply.

A common misconception is to equate throttling with an outage. The difference is fundamental: during an outage, a service doesn’t work for anyone. During throttling, it keeps working — just not without limit for everyone. Anyone who hasn’t yet hit the limit often doesn’t notice anything at all.