Speech-to-Text

Speech-to-text literally means “speech to text”. It refers to software that takes an audio recording of spoken language and turns it into written words. You speak into a phone’s microphone, and the sentence appears as text on the screen. In German, this is also called Spracherkennung (speech recognition). It’s important to note: the program doesn’t understand the content the way a human does — it merely writes down what it heard. Whether the sentence makes sense or what it means is a separate matter.

Why speaking is faster than typing

A practiced person types around 40 words per minute. Speaking is considerably faster, at around 150 words per minute. That’s why doctors, lawyers, and journalists save a lot of time by dictating their reports instead of writing them. In the past, there were typists who transcribed tape recordings by hand. Today, this work is largely taken over by software.

A second reason is accessibility. People who are hard of hearing rely on captions. Manually created captions are expensive, automatic ones are almost free. This is what allows millions of videos to have captions in the first place. For people who have difficulty using their hands, dictation is also often the only comfortable way to produce a text.

Finally, speech-to-text is the gateway to many other AI applications. A voice assistant must first have what you said available as text before it can respond to it. Translation apps that convert spoken sentences into another language also begin with this step. If this first step is flawed, everything that follows is flawed too.

From sound-wave image to sentence

A microphone picks up fluctuations in air pressure and turns them into a long sequence of numbers. This sequence is broken down into very short segments, usually around 20 milliseconds long. For each segment, the program calculates which pitches occur and how strongly. The result can be imagined as an image: a kind of map of the sound over time. Text is then meant to be produced from this image.

This conversion is carried out by a neural network. This is a program that has learned patterns itself from a huge number of examples, rather than following fixed rules. For training, it is given thousands of hours of audio recordings together with the matching written text. In the process, it learns which sound patterns correspond to which sounds and words. Modern systems such as OpenAI’s Whisper were trained on hundreds of thousands of hours of material from the internet.

It is also crucial that the system evaluates speech in context. “Sea” and “see” sound identical but fit into completely different sentences. The program therefore chooses the spelling that is more likely given the sentence context. This is also exactly where the limits become apparent: with dialects, background noise, technical jargon, or several people speaking at once, the error rate rises considerably. This is measured as the word error rate, i.e. the proportion of incorrect or missing words.

Captions, dictation, and call centers

Speech-to-text is most commonly encountered in automatic captions on YouTube and in video conferences. The dictation button on a phone’s keyboard and assistants such as Siri or Alexa also work with it. Messenger apps offer a feature to display a voice message as text when you can’t listen to it at the moment.

In companies, software uses it to analyze customer service phone calls. Common complaints or keywords can be filtered out of the text. Hospitals use dictation systems for medical reports, courts for transcripts. Such applications raise data-protection concerns, since recordings are often processed on servers belonging to third-party providers.

In business news, the term usually appears alongside its counterpart: text-to-speech, conversely, generates an artificial voice from text. Together, both enable conversations with machines, for example in automated telephone hotlines. Anyone who has ever been annoyed while interrupting such a hotline also knows the technology’s weaknesses.