Secret Pattern

Programs are constantly accessing external services: a weather app fetches data from a weather provider, an online shop deducts money via a payment service. So that the external service knows who is making the request, the program needs a kind of digital ID. These IDs are long strings of letters and numbers, and in technical language they are called secrets. A secret pattern is a rule that describes what such an ID typically looks like. Using this rule, a computer can search huge amounts of text and flag every spot that matches the appearance of such an ID. This is how secrets that accidentally ended up somewhere public are found.

A key in the source code is an unlocked house

Developers usually don’t write their program code alone, but upload it to platforms like GitHub. There, teams work on it together, and many projects are even visible to everyone. If someone, out of convenience, writes an access key directly into the code, that key is suddenly public. Whoever finds it can pose as the company to the external service.

The consequences range from costly to existentially threatening. A stolen cloud key is gladly used by attackers to mine cryptocurrency at someone else’s expense. The bill can reach five figures within a few days. Worse is the case where the key grants access to a customer database. Then it’s about the names, addresses, and payment data of real people.

That’s why large platforms and companies automatically scan their code for such findings. Speed is crucial here. Studies show that publicly uploaded keys are often discovered and tried out by automated programs within just a few minutes. A human checking once a week comes far too late.

From prefix to checksum

The simplest form of a secret pattern is a search text with placeholders, technically a regular expression. It describes, for example: first the fixed sequence of letters sk-, then 48 arbitrary letters or digits. Many providers deliberately help by giving their keys a fixed beginning. Such prefixes are unmistakable and make the search very accurate.

It gets more difficult with secrets that have no fixed identifying marker, such as pure random strings or passwords. Here, tools additionally work with the surrounding context: Does the line before it contain the word password or token? And they measure the disorder of the character sequence, the so-called entropy. A real password looks more chaotic than a normal English word. Both together yield a probability, not a certainty.

This is exactly where the typical misconception lies. A match does not automatically mean that a real secret has been found. Sample code in tutorials often contains made-up keys that look exactly the same. Such false alarms are called false positives, and too many of them lead developers to ignore warnings. Better tools therefore check directly with the provider upon a match whether the found key is even still valid.

Where secret patterns show up in everyday work

Most often they run invisibly in the background of development platforms. GitHub automatically scans uploaded code and, upon a finding, informs both the team and the affected provider. The provider then often revokes the key on its own. In addition, there are checks directly on the developer’s machine that block an upload before it happens.

AI has sharpened the issue. Chatbots that help with coding are given entire files and return code. This means secrets can migrate into systems that no one intended for them. Many companies therefore place secret patterns in front of the interface to the AI model and redact detected credentials beforehand.

In the news, you usually encounter the term after a data breach. If it says an attacker gained access via an exposed API key, then either no pattern caught it or no one took the alert seriously. Secret patterns should be distinguished from actual secrets management: the pattern finds mistakes after the fact, whereas a secrets manager prevents them by keeping credentials separate from the code.