Speech-Native

When you talk to a voice assistant, an invisible detour happens with older systems. Your voice is first converted into written text. This text goes to the actual program, which formulates a response. The response is then read aloud by an artificial voice. A speech-native system skips this detour: it takes in the audio recording directly and directly produces sound again. The word essentially means “built from the ground up for speech.”

What gets lost in the detour through text

Written text is a very impoverished form of language. The line “Well, isn’t that great” doesn’t reveal whether someone is thrilled or annoyed. Hesitation, a sigh, or a laugh also disappear without a trace. Likewise, you can’t tell whether a child or an adult is speaking, or whether someone is whispering or shouting. All of this information is contained in the audio recording, but not in the line of text.

A speech-native system retains this information and can react to it. It can itself become quieter, laugh, add emphasis, or insert a pause. For applications like language-learning apps, this is crucial, since pronunciation and stress are what matter there. It also matters for emergency call centers or phone hotlines whether someone sounds calm or panicked.

The second major advantage is speed. Each of the three steps in the old setup costs time, and the delays add up. In the past, it typically took one to two seconds before a response began. Good speech-native systems come in at around a third of a second. That’s roughly the pace at which people take turns speaking in conversation. Only then does a conversation stop feeling like a radio transmission.

From sound signal to response tone

An AI model doesn’t compute with sounds, but with numbers. That’s why the audio recording is first broken down into many small chunks, often several dozen per second. Each chunk is assigned an ID number from a fixed set, similar to letters from an alphabet. These building blocks are called audio tokens. One second of speech thus becomes maybe fifty such building blocks.

The model then works exactly like a text chatbot: it always predicts the next building block. Only here, the building blocks aren’t word fragments, but snippets of sound. At the end, another component converts these snippets back into audible sound. Because the same chain runs from listening all the way to responding, the tone of voice is preserved throughout the entire process.

Such models are trained on very many hours of real conversations. A common misconception is that a speech-native system is simply a chatbot with a better read-aloud voice. The difference lies deeper: the sound isn’t just packaging for the answer, but part of what the model understands and generates. One drawback remains, however. Because no clean text is produced anymore, it’s harder to trace what the system actually understood. Many providers therefore additionally generate a text log on the side.

Voice mode on the phone, hotline on the telephone

The best-known examples are the voice modes of major chatbots, such as those in ChatGPT or Gemini. There, you can interrupt the system mid-sentence, and it stops talking immediately. That was barely possible with the old three-step approach. Translation apps also use this technology to preserve emphasis and vocal tone across language barriers.

In business, the largest market is customer phone contact. Companies use it to replace menus where you have to press numbers. In cars, such systems control navigation and music without requiring the driver to look at a screen. In news and quarterly reports, the term usually comes up when a provider unveils a new model for real-time conversations.

At the same time, the technology is sparking debate. Because voices can be replicated very convincingly, the risk of scam calls using cloned voices is growing. Data privacy also plays a role, since a voice is a biometric feature just like a fingerprint. Many providers therefore technically watermark generated voices or restrict the replication of other people’s voices.