Logging

When a program runs, it often keeps a record in the background of what is happening. It notes, for example: At 2:02 PM, User A logged in. At 2:03 PM, a file was saved. At 2:04 PM, an error occurred. These notes are called a log, sometimes also referred to as a log file. The act of creating such notes is called logging. You can think of it like a ship’s logbook: nobody reads it during normal operation, but after an incident it is the most important source.

Without a log, every error remains a mystery

Software today runs on many computers simultaneously, often distributed across several data centers. If something goes wrong there, nobody can stand by and watch. The error is usually already over before anyone notices it. Only the log preserves what happened at that moment. Without this record, one would have to painstakingly try to reproduce the error, and this often does not succeed.

A second reason is security. If someone gains unauthorized access to a system, they leave traces in the log. Experts can use this to identify which accounts were affected and how far the attacker got. Banks and insurance companies are even legally required to record certain processes in a traceable way. Such especially protected logs are called an audit trail.

With AI systems, a third reason is added. The European AI Act requires that the operation of high-risk applications be automatically recorded. If, for example, software pre-sorts job applications, it must later be possible to verify what data it received and what it output. Logging is therefore not just a technical matter, but a legal requirement.

From timestamp to search field

A single log entry usually consists of three parts. First, the timestamp, accurate to the millisecond. Second, a classification of how important the message is. Third, the actual text with details. There are fixed levels for the classification: from mere additional information through warnings to severe errors. This way, one can later specifically look at only the critical messages.

In the past, logs simply ended up in a text file on the same computer. That is no longer sufficient today, since a service runs on dozens of servers. That’s why all computers send their entries to a central collection point. There, they are made searchable, similar to a search engine. A developer can then filter out, within seconds, all messages related to a specific user from the past week.

The sheer volume is the biggest problem here. Large services generate several terabytes of log data every day. Storage space and analysis cost money, so usually only a portion is kept for a long time. Another common mistake is recording too much: if passwords or health data end up in the log, this itself creates a data protection problem. Good logging is therefore a matter of selection, not completeness.

Logs in everyday life and in corporate reports

On one’s own computer, one encounters logs without noticing. Windows keeps an Event Viewer, Android and iPhone record crash reports. When an app crashes and asks whether it may send a report, it sends an excerpt from its log to the manufacturer. The browser also has a console where websites store their messages.

In the business press, the term usually comes up after outages. After a cloud provider’s failure, a report often appears that reconstructs from the logs which misconfiguration triggered the chain of events. Around logging, an entire industry has also emerged. Providers like Datadog or Splunk earn money by collecting and analyzing log data, and are highly valued on the stock market.

Logging should be distinguished from monitoring. Monitoring continuously observes metrics such as response time or load and raises alarms. Logging, on the other hand, provides the individual events used to explain the alarm. In practice, both work together: monitoring says that something is broken, the log says why.