SPEED-Bench

SPEED-Bench is a testing procedure for computer programs that independently generate texts or answers. Such testing procedures are also called benchmarks: standardized sets of tasks that are presented to different systems in order to compare them. What’s special about SPEED-Bench is already in the name, since “speed” means velocity. What is measured, then, is not only whether a system finds the correct answer, but also how long it takes and how much computing power it consumes. You can think of it like a bicycle test that doesn’t just ask whether the bike arrives intact, but also how fast and with how much effort. It’s important to note: there is no single, universally recognized test of this name in the industry, but rather several procedures that go by similar designations.

Why speed in AI becomes a matter of cost

For a long time, almost only one question mattered for AI systems: How smart is the answer? Well-known tests therefore consist of school-level tasks, programming tasks, or knowledge questions. That makes sense, but it falls short. After all, a system that delivers a perfect answer in two minutes is useless for a chat service with millions of users.

Behind speed lies money, directly. Every answer is computed on expensive specialized chips in large data centers. If a system takes twice as long, operation costs roughly twice as much and also consumes roughly twice as much electricity. Companies that build AI into their products therefore have to weigh things up: a bit more quality against significantly higher running costs.

A second reason is the user experience. People abandon a task when an answer takes too long to arrive. For voice assistants, the threshold lies at a few tenths of a second. Tests that measure speed alongside quality make visible exactly this difference, which remains completely invisible in pure knowledge tests.

What a speed test measures

The procedure is always similar. The system is presented with a fixed list of tasks, always the same one for all candidates. Stopwatches run throughout. In the end, several numbers stand side by side: the success rate on the tasks and various time and consumption values.

Two time values are particularly common. The first is the waiting time until the first character of the answer appears. The second is the output rate, i.e. how many word units follow per second. Such word units are called tokens; a longer English word often consists of two or three of them. In addition, figures on energy consumption or required memory are often included as well.

A typical mistake is to take such numbers as properties of the model. They depend heavily on the hardware on which the test was run, and on the number of simultaneous requests. That’s why results are only comparable if the test conditions are disclosed and identical. Anyone who pits figures from two different sources against each other is usually comparing apples to oranges.

Where such rankings turn up

You most often encounter speed tests in reports about new AI models. Companies then publish tables in which their system performs faster or more economically than the competition. Such figures are marketing and should be read with caution – the company itself chooses which test to show.

They are practically relevant for developers who are purchasing an AI service. Providers usually sell access per million tokens, and on top of that come commitments regarding response times. Speed tests help in deciding which provider fits which purpose. For a customer-service chatbot, speed counts; for an overnight data analysis, price matters more.

SPEED-Bench should be distinguished from pure capability tests such as MMLU, a large knowledge quiz for language models. There, it’s solely about correctness. Only the combination of both gives a useful picture of whether a system is suitable for a real product.