Speech-to-Speech Model

A speech-to-speech model is a computer program that listens to spoken language and answers by speaking itself. What’s important here is what does not happen: the spoken input is not translated into written text in between. Older voice assistants worked in three separate steps. First, a program transcribed what had been said. Then a second program came up with an answer in text form. A third program finally read this text aloud. A speech-to-speech model handles all three tasks within a single system that works directly with sound.

What gets lost in the detour through text

Written text is a surprisingly impoverished version of what someone says. The sentence “Well, you really did a great job there” can be honest praise or pure mockery. On paper, both look identical. Anyone who hears it recognizes the difference instantly from the tone of voice. A system that only passes the text along discards this information without any replacement.

A speech-to-speech model retains the sound. So it can notice that someone is hesitating, sounds irritated, or is whispering. And it can respond in kind: more quietly, more quickly, with a follow-up question conveyed through tone rather than just a question mark. For applications like language-learning apps or phone hotlines, this makes a noticeable difference.

The second advantage is speed. In the old three-step chain, each step waits for the previous one to finish. Together, this often adds up to a pause of one to two seconds. That may not sound like much, but it destroys any sense of natural conversation. On average, humans respond to each other after about two-tenths of a second. Modern speech-to-speech systems come noticeably closer to that value.

From sound-wave snippet to audio building block

To a computer, sound is initially just a long series of numbers. For a model to be able to work with it, the recording is broken down into tiny segments, often just a few hundredths of a second long. Each segment is assigned a kind of number from a fixed catalog of sound building blocks. These building blocks are called audio tokens. One second of speech thus becomes maybe fifty such building blocks.

After that, something happens that’s familiar from chatbots: the model predicts, block by block, what should come next. Except here, these are sound building blocks instead of words. From the predicted chain, another component, the vocoder, reconstructs audible sound. The whole system is trained using huge amounts of recorded conversations.

A common misconception is that the model no longer has any understanding of text. That’s not quite true. Many systems are additionally trained on text or output a text track alongside the audio. The difference is that text is no longer the sole bottleneck that everything must pass through.

Where these systems speak today

The best known examples are the voice modes of major chat apps. There, you can hold a conversation, interrupt the model, and it adapts. Translation devices also use this technology: you speak German into the device, and Spanish comes out, sometimes even in a voice resembling your own.

In business, the largest market is telephone customer service. A conversation with a human costs a company several euros, while an automated one costs only a fraction of that. That’s why speech-to-speech models show up in quarterly reports and stock market news. Car manufacturers and makers of household appliances are also building them in.

As adoption spreads, the problems grow too. If a system can imitate any voice after just a few seconds, phone fraud becomes easier. EU lawmakers therefore require that artificially generated voices be recognizable as such. As a result, you’ll often encounter this term twice in the news: as a product announcement and as a security topic.