GIFT-Eval

Much data arises in step with time: a city’s hourly electricity consumption, a store’s daily revenue, the number of cars on a street per minute. Such measurement series are called time series. Computer programs try to estimate the next values of such a series from its past. GIFT-Eval is a test that compares exactly these programs against each other. It gives everyone the same tasks from around two dozen data collections and measures who deviates the least. It was developed at the software company Salesforce, and the results appear in a freely accessible leaderboard on the internet.

Why forecasting software needs a common test bench

Before GIFT-Eval, almost every research team tested its method on its own data. This led to a well-known problem: everyone picked the examples where their own idea looked good. Comparisons between two publications were thus practically worthless. A shared test bench with fixed rules puts an end to this confusion.

There is also a second problem that has only become important in recent years. There are now so-called foundation models for time series. These are large models that were trained once on huge amounts of data and are then supposed to forecast any new series without learning again. Whether this really works can only be checked if the test data was previously unknown to the model. GIFT-Eval explicitly takes care to keep this separation clean.

This is practically relevant because money depends on such forecasts. Grid operators use them to plan power plant deployment, retailers to plan their inventories, airlines their crews. One percentage point less error can mean millions there. Buyers don’t want to have to trust a vendor’s marketing promise.

How the test is structured and measured

The foundation consists of around 24 data collections from seven domains, including energy, transportation, weather, sales, and finance. This produces more than 90 individual test cases with a total of several hundred million data points. The cases deliberately differ greatly: some series are measured every second, others only once a year. The forecast horizon also varies, sometimes involving just a few steps, sometimes very long horizons.

Evaluation is done using error measures that gauge the distance between the forecast and the actually observed value. A second measure checks how well a model assesses its own uncertainty. After all, a good forecast doesn’t just state a number but also how confident it is. So that datasets of vastly different scales can even be combined, the errors are related to a very simple comparison method. This reference essentially predicts that the last time segment will repeat.

In addition, there is a second, considerably larger dataset intended only for training. Anyone developing their own model is meant to learn on it and be evaluated only on the test data. This separation is the crux of the matter. Without it, it would be impossible to tell whether a model truly understands patterns or has simply memorized the solution.

GIFT-Eval in trade news and product announcements

The term is encountered most often in announcements from technology companies. When Google, Amazon, Salesforce, or a startup introduces a new time series model, its ranking on the GIFT-Eval leaderboard is often already in the headline. Phrases like “ranked first on GIFT-Eval” have become a selling point. The leaderboard itself is maintained on the Hugging Face platform and is accessible to everyone.

In companies, the test comes up when software for sales or demand planning is being selected. It provides a first, neutral point of reference there. However, one should not rely on it alone. A model that leads on average across more than 90 test cases can still perform poorly on a particular company’s specific data.

A common misconception is also that GIFT-Eval says something about language models like ChatGPT. It does not. This is exclusively about numerical series over time, not about text. And as with any public leaderboard, the rule applies: the more well-known it becomes, the more developers specifically optimize toward it. This is precisely what weakens its informative value in the long run.