Voice Cloning

Voice cloning literally means cloning a voice. A computer program listens to recordings of a real person and creates an artificial copy of their voice from them. This copy can then speak sentences the real person never said. You type in a text, and out comes an audio file in the cloned voice. In the past, this required hours of studio recordings. Today, good systems only need a few seconds of material, for example from a video or a voice message.

From audiobook production to the grandchild scam

The benefits are obvious. An audiobook can be released in several languages without the narrator having to go back into the studio. Films are dubbed while still retaining the actor’s familiar voice. People who lose their voice due to illness can create a copy of it beforehand and continue speaking with it later. For them, the technology is not a toy, but a piece of identity.

But the same technology is also a tool for fraud. Criminals call, sound like a family member, and urgently ask for money. This scheme is a modern variant of the grandchild scam. In several cases, companies have also suffered losses because a supposed boss’s voice on the phone ordered a wire transfer. For decades, voice was considered proof of authenticity. That proof no longer holds.

On top of that come legal questions. Dubbing actors and musicians fear that their voice will be copied without permission and resold. In the EU, the AI Act, the European Union’s AI law, requires that artificially generated media content be labeled. Whether this can be enforced in practice remains an open question.

What the model reads out of just a few seconds

At its core, two components work together. The first listens to the sample recording and compresses it into a list of numbers. This list describes what characterizes the voice: pitch, timbre, speaking pace, accent. Experts call this a voice fingerprint. The content of the recording doesn’t matter here, only the sound.

The second component is a speech generator. It receives the desired text and the voice fingerprint. From these, it generates an audio track step by step. It was previously trained on thousands of hours of speech from many different people. In doing so, it learned how letter sequences sound in German and where the stresses fall. For a new voice, it therefore doesn’t need to learn from scratch, but only has to apply the fingerprint.

A common misconception is that word snippets from the original recording are simply pasted together here. That was the old technique, and it sounded tinny. Modern systems compute the audio track completely anew. That’s why they can also pronounce words that never appeared in the source material. Limits remain nonetheless: strong emotions, screaming, or singing often come out less convincingly than calm reading.

Cloned voices in apps, advertising, and headlines

Cloned voices are most commonly encountered on video platforms. Many short clips are narrated by an artificial voice, sometimes in the voice of well-known personalities. Commercials and podcasts also use the technology to record corrections without having to book the narrator again. Providers like ElevenLabs or the voice features of major chatbots have made this mainstream.

In the news, the term usually comes up in connection with deepfakes. A deepfake is a fabricated piece of media content designed to look genuine. Before elections, fake calls and audio clips of politicians have already circulated. Banks, in turn, are overhauling their systems because voice as a password over the phone has become unreliable.

Anyone wanting to protect themselves should know a simple rule. If faced with a surprising demand for money over the phone, hang up and call back on the known number. Families sometimes agree on a code word for emergencies. Technical detection programs for fake audio exist, but they are unreliable. The race between forgery and detection has not yet been decided.