Diarization

When multiple people speak into a recording, a single continuous audio track is created. Diarization is the process that breaks this track into segments and assigns a speaking person to each segment. The result is a list with timestamps: from second 0 to 12, Person A speaks, then until second 20, Person B speaks. It is important to understand what diarization does not do: it does not transcribe the content, nor does it know the names of the people involved. It only distinguishes the voices from one another and then simply labels them Speaker 1 and Speaker 2. Only in combination with speech recognition, which converts spoken words into text, does this become a readable conversation transcript.

Why a transcript without speaker attribution is of little value

An automatically generated transcript without speaker changes is a wall of text. You read sentences but don’t know who made which statement. In a discussion with five people, such a text is practically useless. This is exactly the gap that diarization fills, which is why it is built into almost every transcription tool.

In some fields, even more than readability depends on it. In a medical consultation, it must be clear whether a statement comes from the doctor or the patient. The same applies to court transcripts or interviews in journalism. An incorrect attribution distorts the meaning, even though not a single word is misspelled.

Then there is the economic aspect. Companies analyze phone calls with customers to identify complaints or train employees. To do this, one must be able to separate the advisor’s share of the conversation from the customer’s. Without diarization, such an analysis cannot be automated.

From voice timbre to speaker change

The first step is segmentation. The system searches for points in the audio where the sound changes noticeably and cuts there. This creates short pieces of a few seconds each, which presumably each contain only one person. Silence and background noise are filtered out in the process.

Afterwards, each piece is converted into a series of numbers that describes the characteristics of the voice. Such series of numbers are called embeddings. You can imagine them as a voice’s fingerprint: pitch, timbre, and manner of speaking are turned into a point in a coordinate system. Pieces from the same person end up close together, while pieces from different people end up further apart.

In the final step, an algorithm groups the points into clusters. Each cluster represents one person. Often the system does not know in advance how many people are actually speaking and must also estimate this number. The most difficult case is simultaneous speech: when two people interrupt each other, the voice fingerprints overlap and the assignment becomes unreliable. Very short interjections, such as an affirmative “yes,” are also frequently missed.

Diarization in meeting tools and subtitles

This process is most commonly encountered in video conferencing programs. When Teams, Zoom, or Google Meet provide a summary with names after a meeting, diarization is behind it. There, the task is relatively easy because each person has their own microphone. The program knows the logged-in accounts and can therefore display real names instead of Speaker 1.

It becomes significantly more challenging with a single recording from a room, such as a podcast or an interview recorded with a phone. Providers of transcription services explicitly advertise speaker separation as an additional feature here. Subtitles for television programs also use it to attribute dialogue to different people.

In news about AI, the term usually appears together with speech recognition models like Whisper. Such models only provide the text on their own and are then combined with a separate diarization component. A common misconception is therefore to consider speech recognition and diarization to be the same thing. They are two separate tasks that, only when combined, produce a complete conversation transcript.