Threat Modeling

Threat modeling roughly means “playing through threats.” A team sits down and asks itself: Who could attack our system? What would this person gain from it? And at what point would the attack be easiest? The answers are written down, sorted by danger level, and then a decision is made about what to protect against. Timing matters here: ideally this happens before the software is built, not only after the first damage has occurred.

Why Guessing Is More Expensive Than Thinking

Security costs money, time, and convenience. You can’t secure everything at once. Without an organized analysis, gut feeling often decides, and when it comes to security, gut feeling is frequently wrong. Teams then build elaborate protective mechanisms at a spot no one ever attacks, while leaving the open door right next to it unguarded.

On top of that comes the cost factor. A design flaw can be fixed at the whiteboard in half an hour. The same flaw in a live application with millions of users costs weeks, because data has to be migrated, interfaces changed, and old versions retired. If the flaw is only discovered through an actual breach, reporting obligations, fines, and loss of trust are added on top.

For AI systems, this topic has become significantly more important in recent years. A language model that reads emails and independently takes actions has attack surfaces that classic software doesn’t know. Anyone who puts such systems online without a prior threat analysis will only discover the gaps through the attackers.

Four Questions, One Diagram, One Priority List

In practice, threat modeling usually runs through four questions. First: What are we actually building? For this, the team draws a simple diagram with all the components, data flows, and outer boundaries. Second: What can go wrong? Third: What are we doing about it? Fourth: Did we do a good enough job?

For the second question, there are checklists so that no one overlooks the obvious attacks. The best-known one is called STRIDE and names six categories: impersonating someone else, altering data unnoticed, later denying an action, capturing confidential information, disabling a system through overload, and gaining more privileges than allowed. The team goes through the diagram component by component and checks each category.

After that, sorting takes place. Every threat found gets a rough assessment: How likely is it, and how large would the damage be? Only what lands at the top gets secured. The rest is deliberately accepted and documented. This is exactly what distinguishes threat modeling from a penetration test: the penetration test looks for real gaps in finished software, while the threat model looks for design flaws in thinking.

From Bank Servers to AI Assistants

Banks, hospitals, and government agencies have been doing threat modeling for years, often because regulators require it. In the software industry, it’s a fixed part of the process for every new product at large vendors. Microsoft shaped the approach heavily in the 2000s and still provides a free tool for it today.

In tech news, the term currently appears mainly in connection with AI agents. These are programs that use a language model to complete tasks independently, such as searching the web or editing files. A typical attack scenario: a webpage contains hidden text that gives the assistant new instructions. Such attacks are only found if one has systematically thought beforehand about what data the system reads in from outside.

A common misconception is that threat modeling is a one-time document for the filing cabinet. It only makes sense if it grows along with the system. If a new interface is added, the attack surface changes, and the model must be adjusted.