Systemic Risks

Some harms remain small and localized. If a translation program mistranslates a word, one person gets annoyed. Other harms, however, spread and drag further areas along with them. It is precisely this second kind that is meant by systemic risks: dangers that ripple through an entire system, that is, a society, a market, or the shaping of public opinion. The term originally comes from the world of finance, where a single collapsing major bank can drag entire economies down with it. Today it is used in exactly the same way for computer programs that autonomously generate texts or images.

Why lawmakers are looking especially closely here

The difference between an ordinary risk and a systemic risk is not the severity of a single case. It is the reach. A program used by millions of people at once makes the same mistake millions of times over. And a mistake that many people make at the same time is harder to correct than isolated slip-ups.

The European Union has incorporated this idea into law. The AI Act, Europe’s AI law, has its own category for especially powerful foundation models. A foundation model is a large, generally trained AI system on which many applications are built. If such a model reaches a certain level of computing power used in training, it automatically counts as a model with systemic risk. Additional obligations then apply to its providers.

The Digital Services Act, the EU’s law on digital services, works in a similar way. Platforms with more than 45 million users in the EU must investigate their own systemic risks and report on them. Size alone is therefore enough to trigger an oversight obligation here.

How to recognize a systemic risk

The laws name several typical areas. These include the manipulation of public opinion, for instance through mass-produced disinformation ahead of an election. They include the discrimination against entire population groups, when a model systematically produces skewed judgments. And they include the danger that someone might misuse a model to compile dangerous knowledge, for example about weapons or attacks on computer networks.

To find such risks, providers rely on, among other things, red teaming. In this process, a dedicated team deliberately attacks the model and tries to provoke unwanted responses from it. Whatever comes to light in the process is documented and is meant to be mitigated before release. On top of this come reporting obligations for serious incidents as well as requirements for safeguarding the models against theft.

A common misconception: systemic risk does not mean that a model is banned. It only means that stricter rules apply. The AI Act provides for bans separately from this, for example for state systems that score the social behavior of citizens.

The term in news and business reports

Anyone who reads business news encounters the expression regularly. When a company unveils a new large language model, the question of whether it falls under the stricter category almost always comes up. Costs depend on this: additional testing, documentation, and staff for oversight.

The term also appears in the annual reports of publicly listed companies. There it is usually about which regulatory costs the company is facing. This is of interest to investors because strict requirements burden smaller providers more heavily than large corporations. Some experts even see this as an advantage for market leaders.

Systemic risk should not be confused with the term high-risk AI from the same law. High-risk refers to a specific use case, for example AI used in candidate selection or in medical devices. Systemic risk, by contrast, refers to the sheer capability of a general-purpose model, regardless of what it is later used for.