XGBoost

XGBoost is a program that computes predictions from data arranged in tables. Such a table might, for example, contain customers: one customer per row, one attribute such as age, income, or place of residence per column. The question might then be: Will this customer repay their loan? XGBoost answers this by setting up a very large number of small yes-no rules and combining their individual judgments into an overall verdict. Each single rule is crude and often wrong, but the sum of thousands of them is astonishingly accurate. The name stands for “eXtreme Gradient Boosting” and describes the computational method behind it. The program is freely available and was released in 2014 by the computer scientist Tianqi Chen.

The quiet winner of tabular data

The news is almost always about language models like ChatGPT. But most of the data that companies actually work with sits in tables: revenues, transactions, measurements, customer data. It is precisely there that XGBoost is often better than a neural network, that is, better than the technology behind the well-known AI chatbots. This has remained true to this day, even though many researchers had expected otherwise.

A second reason is cost. XGBoost trains on an ordinary laptop in minutes, whereas a large language model needs weeks on expensive specialized hardware. Anyone building fraud detection for a bank can create a working model with it in an afternoon. This lowers the barrier to entry considerably.

Its strength became especially visible on Kaggle, a platform where companies run data analysis competitions. Over the years, a striking number of teams there won using XGBoost or a very similar method. That made the program the standard tool in the industry.

A thousand trees correcting one another

The basic building block is called a decision tree. It is a chain of simple questions: Is income above 40,000 euros? If so, is the customer older than 30? At the end of each chain of questions there is an estimate. A single such tree is imprecise because it simplifies reality drastically.

The trick is called boosting. XGBoost builds the first tree, then looks at where it went wrong, and builds a second tree that specifically fixes those errors. The third tree corrects the remaining errors of the first two, and so on. It is like a series of proofreaders: each one catches the mistakes that all the previous ones missed. After a few hundred rounds, hardly anything is left uncorrected.

It is important to distinguish this from Random Forest, a related method. There, all trees grow independently of one another and vote at the end. With XGBoost, each tree depends on the result of the previous ones, so the trees are built one after another. This is more accurate, but also more prone to the model simply memorizing the training data. To counter this memorization, XGBoost contains several built-in brakes that keep the trees small and the corrections cautious.

Credit scoring, shopping baskets, and wind power

Banks and insurers use such models for risk assessments. When a bank decides within seconds whether you get an installment loan, a method of this kind is often behind it. Credit card companies also use it to check every payment against whether it fits your usual behavior.

In online retail, XGBoost estimates how much of which item should be in stock next week. Energy providers use it to forecast how much electricity a wind farm will deliver tomorrow, based on weather data from previous years. In medical research, it serves to estimate disease risks from lab values.

In job postings for data analysts, XGBoost almost always appears alongside Python, the standard programming language for data analysis. Related programs with similar technology are called LightGBM and CatBoost. A common misconception is that XGBoost has been superseded by modern AI. For tabular data, that is simply not the case.