Detection Engineering

Large companies run thousands of computers, servers, and programs. All of them constantly write logs: who logged in, which file was opened, which connection went out to the internet. These records are called logs. Detection Engineering is the work of automatically fishing the suspicious activity out of this mass of logs. To do this, specialists write rules that describe what an attack looks like in the log data. If an event matches a rule, it triggers an alert that a human then reviews.

Why alerts alone don’t mean security

Attacks rarely fail because nobody could have seen them. They fail because the traces get lost in the noise. A mid-sized company can easily generate billions of log lines per day. No human reads that. Without good rules, the decisive entry simply goes unnoticed.

The other extreme is just as dangerous. Filtering too coarsely produces hundreds of alerts per shift, almost all of them harmless. Specialists call this alert fatigue. After the fiftieth false alarm, the security team clicks away the real one too. Detection Engineering therefore tries to keep the number of alerts low and their hit rate high.

There is also an economic point. Detection is significantly cheaper than remediation. An attack that is noticed after two hours costs a company a fraction of what one that goes unnoticed for months would cost. This is precisely why the role of the Detection Engineer has become its own job profile in recent years.

From attack idea to tested rule

At the beginning stands not the rule, but the attack behavior. The team asks: how does an attacker concretely proceed when reading passwords out of working memory? MITRE ATT&CK, a public collection of known attack techniques, usually serves as the catalog. From such a technique, one derives which trace it leaves behind in the logs.

From this, a rule is formulated, often in a standardized format such as Sigma or YARA. A simple rule might read: an Office program launches a command-line window. For a text document, this is unusual; for malware in an attachment, it’s typical. Such rules then run in a SIEM, a system that centrally collects and searches all logs.

What comes afterward is crucial. The rule is tested against real attack simulations, stored in the version control system, and provided with documentation. In other words, it is treated like program code, not like a setting someone configures once. As soon as software, network, or attack methods change, the rule has to be updated accordingly. A common misconception is that rules run forever: without maintenance, they lose their effectiveness within months.

Where these rules take effect in everyday life

This work usually only becomes visible indirectly. If your school blocks access to an account because someone logged in from another country, there is a detection rule behind it. Your bank’s warning about an unusual payment follows the same principle. The difference from a virus scanner is important: a scanner looks for known malicious files, Detection Engineering looks for suspicious behavior.

In the news, this topic almost always comes up after a major hacking attack. It is then said that the attacker went undetected in the network for weeks. Translated, that means: there was no matching rule for this particular approach, or its alert got lost.

Artificial intelligence is currently changing the field from two sides. Language models now help with drafting rules and pre-sorting alerts. At the same time, attackers use the same tools to vary their methods more quickly. Providers such as Microsoft, CrowdStrike, or Elastic therefore sell not just software, but above all maintained rule collections as an ongoing service.