AA-WER Streaming Benchmark

There are programs that automatically convert spoken language into text. Some wait until a person has finished speaking. Others write along while speech is still ongoing — this is called streaming. The AA-WER Streaming Benchmark is a standardized test for the second kind. It has many systems transcribe the same audio recordings and then counts the errors. The result is an error rate in percent, which allows providers to be compared directly.

Why an error rate alone is not enough

The common metric for speech recognition is called Word Error Rate, or WER for short. It indicates how many words per hundred are wrong, missing, or extra. A value of 5 percent means: one in twenty words is incorrect. This figure is simple and has been established for decades.

For streaming systems, however, it can easily be misleading. Such systems often correct themselves. First, a preliminary word appears, and shortly afterward the model replaces it with a better one. If only the final text is measured, the error rate looks excellent. Yet the user has still read nonsense on the screen for several seconds.

This is precisely the gap that the AA-WER Streaming Benchmark closes. The “AA” stands for the evaluating body that defines the test data and rules. The important part is the second half of the name: what is evaluated is the text as it emerges live. A system that constantly jumps back and forth performs worse here than one that writes along calmly and stably. This is inconvenient for providers but very useful for buyers.

From audio archive to percentage figure

The basis of every such test is a collection of audio recordings with verified transcripts. These transcripts were created by humans and are considered the correct solution. The test does not play back the recordings as a single file, but in small chunks, the way a microphone would deliver them. The system must therefore work under the same conditions as in real operation.

The evaluation program then compares the machine output word by word with the human transcript. It counts three types of errors: omitted words, invented words, and confused words. The sum is divided by the total number of words. The result is the Word Error Rate.

In addition, there are metrics that only exist for streaming. Latency describes how many milliseconds pass between the spoken word and the written word. Stability describes how often text already displayed is subsequently changed. A good test report states all three figures side by side, because they influence one another. Anyone who wants to lower the error rate has to wait longer — which worsens latency.

Who reads such figures and for what purpose

In everyday life, streaming speech recognition is used in many places. Automatic captions in video conferences are one example, as are live captions on video platforms. Dictation functions on mobile phones and voice assistants in cars also work this way. In all of these, it is immediately noticeable when a system is jittery.

In news reports about technology companies, benchmark figures come up regularly. A provider might announce, for instance, that its new model achieves a Word Error Rate of 4.2 percent instead of the previous 6.1 percent. Such announcements are only meaningful if the test conditions are stated. What matters is the recordings used, the language, and the permitted delay.

A common misconception is that a low error rate automatically means a better product. Test recordings are usually cleanly captured and clearly spoken. In a noisy open-plan office or with a strong dialect, the values quickly rise. Benchmarks like this one therefore provide a fair comparison, but no guarantee for one’s own use case.