Tracing

When you press a button in an app, rarely does just one thing happen in the background. Your request often travels through a dozen programs running on different machines. Tracing means: the system records precisely, for every single request, which path it took. For each intermediate step, it logs when it began, how long it took, and whether it succeeded. In the end, this produces a kind of travel log for a single request. This exact log is called a trace.

Why an error log alone isn’t enough

In the past, programs simply wrote text lines to a file, so-called logs. As long as a program ran alone on a single machine, that was enough. Today, however, a service like an online shop consists of many small programs. Each of them writes its own lines. When an order fails, the clues are scattered across twenty files.

Tracing solves exactly this problem. Every request is given a unique number when it enters the system. This number is passed along to every program involved. Afterwards, all the individual messages can be reassembled into a single story. Twenty disconnected files become one coherent sequence of events.

The second reason is speed. Users notice when a page takes three seconds to load, but nobody immediately knows why. A trace shows that two of those seconds were consumed by a single database query. Without this breakdown, developers can only guess. Large providers use such measurements to justify whether an optimization is even worthwhile.

Spans, parents and children

A trace consists of many small building blocks called spans. A span is a single work step with a start time and an end time, for example a database query. Spans can be nested within one another. The call to the payment service is then the parent, and the request to the bank is the child. This creates a tree that maps out the sequence of events.

For this to work across machine boundaries, each program passes the trace number on to the next. This handoff is called context propagation. A widespread standard for this is called OpenTelemetry, and it is supported by many providers. The result is usually displayed as a bar chart. Each bar is a span, and its length is the duration.

A common misconception: that tracing captures every single request. With millions of requests per minute, that would be far too expensive. That’s why usually only a fraction is recorded, say one percent. This is called sampling. Failed requests are often given priority for recording.

From the data center to the AI chatbot

Tracing is most visible in large online services. When a streaming service or a bank goes down, technicians search for the cause in such recordings. Companies like Datadog or Grafana make their money exactly from this. In financial reports, this field appears under the umbrella term observability, meaning the observability of systems.

Since the boom of language models, tracing has found a second home. An AI assistant often looks something up first, then calls a tool, and queries the model multiple times. Without a trace, it’s nearly impossible to tell at which point a wrong answer originated. Tools like LangSmith therefore record every intermediate question and every answer. They also show the costs incurred per step.

For you personally, this runs invisibly in the background. Nevertheless, it’s the reason why outages today are often fixed in minutes rather than days. Anyone who later gets into programming will encounter this term by their first larger project at the latest.