Speech-to-Speech

Anyone who wanted to talk to an AI in the past had to accept a long detour. The system first converted the speech into text, translated that text, and then read the resulting text back aloud. Three steps, three possible sources of error. Speech-to-Speech turns this into a single step: spoken input goes in, spoken output comes out. The intermediate stage of text is eliminated. In the process, the output audio can be in a different language, sound in a different voice, or even imitate the tone of the original.

Why the direct route makes a difference

Speech contains more information than text. When someone asks a question, you can hear whether they sound uncertain. When someone makes a joke, you can hear the wink in their voice. If you write that sentence down, all of that gets lost. The classic three-step approach — speech to text, translate text, read text aloud — inevitably throws this information away.

Speech-to-Speech can preserve these nuances because it works directly on the audio signal. This is especially crucial in real-time conversations. Delays of several seconds — typical of the three-step approach — interrupt the natural flow of conversation. A Speech-to-Speech system, by contrast, can respond significantly faster. That’s the difference between a clunky interpreter phone call and an almost normal conversation.

How a Speech-to-Speech model is built

At its core is a neural network — a system of many interconnected computing units that learns from vast amounts of audio data. It doesn’t learn what a word means, but rather how sounds, stress, and pauses belong together. The input is converted into a compact representation, which can be thought of as a kind of acoustic fingerprint. From this fingerprint, a second part of the network then generates the output audio.

So that the model doesn’t merely copy speech but processes it meaningfully, it is trained on millions of hours of audio material. In the process, it learns which sound sequences in one language typically follow which sequences in another language. Some systems additionally learn to clone voices: they analyze a few seconds of a voice and then generate the output in exactly that timbre. Others learn to recognize emotions and preserve them in the target language.

An important difference from classic translation systems: Speech-to-Speech models don’t need to understand what a word means — at least not in the same way a human does. They recognize patterns. This is a strength in terms of speed and naturalness, but it can become a weakness when technical jargon or strongly divergent dialects come into play.

Speech-to-Speech in products and news

In May 2024, OpenAI unveiled GPT-4o, a model that can process speech directly and respond directly — without the text intermediate step described above. In the live demo, the system responded in under 300 milliseconds, roughly matching a human’s response time in conversation. This was the most prominent public demonstration of the technology to date.

In everyday life, Speech-to-Speech is mainly encountered in real-time translation apps and conferencing tools designed to interpret simultaneously. Headphone manufacturers like Google are working on delivering simultaneous translation directly into the ear — the speaker talks in English, the listener hears German, with no noticeable pause. Speech-to-Speech systems are also used in customer service, translating calls into other languages at the push of a button.

One topic that keeps coming up in this context is misuse. Because some systems can clone voices with deceptive realism, they can be used to create fake audio recordings. That’s why a dedicated field of research is devoted to detecting whether an audio clip is genuine or AI-generated. Speech-to-Speech is therefore not just a communication technology — it also has a security dimension.