Gradient Boosting

Gradient Boosting is a method by which computers learn to make predictions from example data. Suppose one wants to estimate the price of an apartment from size, location, and year built. A single simple computational model usually only manages this roughly. Gradient Boosting therefore builds very many simple models, but not simultaneously — one after another. Each new model looks at where the previous ones went wrong and tries to correct exactly that deviation. The sum of all steps ultimately produces a surprisingly accurate estimate.

The quiet winner with tabular data

The news is almost always about large language models. In many companies, however, something else is working in the background. As soon as data is arranged in a table with columns and rows, Gradient Boosting is often the best choice. Banks use it to estimate credit risks, insurers calculate claim probabilities, and online shops predict sales figures.

The reason is quite plain: with such data, Gradient Boosting regularly beats neural networks, despite requiring much less computing power. A usable model can be built on an ordinary laptop within minutes. A neural network would need more data for this and usually also special graphics cards.

In competitions on the data platform Kaggle, where participants compete for the best predictions, Gradient Boosting has won almost all tabular data tasks for years. This is a good indication that modern AI does not always have to consist of enormous models.

Step by step against one's own error

The simple models are almost always decision trees. A decision tree is a chain of yes-no questions, such as: Is the apartment larger than 80 square meters? At the end of each chain of questions stands a number as the estimate. A single such tree is deliberately kept small and imprecise.

The first tree delivers a raw estimate. Afterward, the error — that is, the difference from the truth — is calculated for each example. The second tree now learns not the prices, but precisely these errors. Its prediction is added to the first. This is repeated hundreds or thousands of times.

The word gradient describes the direction in which the error can be reduced fastest. So that the corrections do not overshoot, each new tree is only factored in with a small proportion, often five or ten percent. One can imagine this like adjusting a telescope: many small turns lead to a sharp image, one large turn shoots past the target. The difference from Random Forest, a related method, is important here: there, all trees are created independently of one another and vote at the end. With Gradient Boosting, each tree depends on the result of the previous one.

XGBoost, LightGBM, and everyday life

Most often, one does not encounter the term directly, but through the names of well-known software libraries. XGBoost, LightGBM, and CatBoost are all implementations of Gradient Boosting. They are freely available and are used by companies worldwide. Anyone reading a job posting for data analysis will find these names there very frequently.

In everyday life, the method operates in inconspicuous places. It helps decide whether a credit card payment is flagged as potential fraud. It ranks search results and product recommendations. Price predictions for flights or ride-hailing services also often run through such models.

A typical misconception: because the method consists of trees with clear yes-no questions, it seems easily understandable. With a thousand trees, however, this is no longer the case. That is why there are additional tools that show which inputs particularly influenced a decision. Especially in credit decisions, regulators demand such explanations.