Capability Threshold

Companies that build large AI systems impose rules on themselves. One of them states: beyond a certain level of capability, a system becomes dangerous enough that it can no longer simply be distributed as usual. This exact boundary is called a capability threshold. It is set down in writing before the system is even finished. Afterward, every new system is tested to see whether it reaches the boundary. If it does, previously agreed-upon measures take effect.

Why a limit is drawn in advance

The point lies in the timing. Anyone who only considers whether a system is too powerful after release is deciding under pressure. By then there are already users, customers, and competitors. If the limit is instead set months beforehand, the decision is more level-headed. In a sense, the company ties itself to the mast before the sirens start singing.

Concretely, this concerns capabilities with high potential for harm. Typical examples include: reliably helping a layperson produce biological weapons, independently carrying out severe cyberattacks, or self-replicating and evading control. Such capabilities should not be noticed only once someone has already exploited them.

It is important to distinguish this from the risk itself. A capability alone is not yet harm. A model that explains chemistry very well causes no damage as long as safeguards are in place. The threshold therefore only describes the point at which the effort put into protection must increase – not the point at which a system is banned.

How it is tested whether the threshold has been reached

Before release, a model undergoes tests known as evaluations. Experts deliberately give it tasks from the dangerous domain. External teams are often involved as well, trying to coax the critical answers out of the system. This deliberate attacking is called red teaming. The results reveal whether the previously defined limit has been exceeded.

The difficult part is measurability. A phrase like “dangerously good at biology” cannot be tested as such. That’s why the threshold is translated into concrete tests: a certain success rate on defined tasks, or a comparison with human experts. Some providers additionally work with tiers reminiscent of safety levels in a laboratory. Each tier comes with its own requirements.

Once a threshold is reached, graduated consequences follow. These can include stricter filters, access limited to vetted customers, better protection of model weights against theft, or a delay in release. A common misconception is that a crossed threshold automatically means the end of a model. Usually it just means: not like this, not for everyone, not right now.

Where the term shows up in the news

Major AI providers have published their own safety frameworks containing such thresholds. At Anthropic, the document is called the Responsible Scaling Policy; at OpenAI, the Preparedness Framework; at Google DeepMind, the Frontier Safety Framework. When a new model is released, companies typically report which tier it has reached. Such announcements are of interest to investors because they indicate how freely a product may be sold.

Legislators have also picked up on the idea. The European Union’s AI Act includes a threshold for models with particularly high risk, beyond which additional obligations apply. The difference from corporate rules lies in enforceability: a company can change or water down a voluntary commitment, but not a law.

This is exactly where the criticism begins. The party that sets, measures, and evaluates the threshold is often the very same company that wants to sell the product. Moreover, the tests are still young and incomplete. A model may appear harmless during testing and later perform far better with improved prompting. Capability thresholds are therefore a useful tool, but no guarantee.