Speaker Diarization

In a recording with multiple people, there are two different questions. The first is: What words were said? The second is: Who said them? Speaker Diarization answers only the second question. To do this, a program breaks the audio track into short segments and groups them by voice. The result is a list of time spans with labels like “Speaker A” and “Speaker B”. The system does not know the real names – it only distinguishes that two different people are speaking here.

Why a transcript without speaker attribution has little value

Modern speech recognition produces remarkably good text. But without speaker attribution, all that results is a wall of text. In an interview, you then don’t know which statement came from the journalist and which from the guest. This information is often the single most important part of the entire conversation.

This becomes especially clear in professions where records carry legal weight. In doctor’s appointments, court hearings, or customer calls, it must be clear who made a commitment. Companies that analyze sales calls also need this separation: they want to know how much talk time the customer had. A metric like “advisor’s share of conversation” cannot be calculated at all without diarization.

A second reason is data preparation for AI systems. Anyone training a language model on thousands of hours of podcasts needs clean, individual voices. Diarization is a preparatory step for this that runs in the background. Errors at this stage propagate into all later results.

From voiceprints to groups

The first step is voice activity detection. It marks the points at which speech actually occurs and cuts out silence, music, and noise. After that, the speech is broken into short pieces of a few seconds each. For each piece, a neural network – a trained computational model – calculates a kind of sequence of numbers that describes the sound of the voice.

This sequence of numbers is called an embedding, and it can be understood as a voiceprint. Two segments from the same person produce similar sequences of numbers, while two different people produce clearly different ones. In the final step, the program sorts all segments into groups by similarity. Each group is then one speaker. Whether there are two groups or seven, the system often has to estimate itself.

The most difficult case is overlapping speech. When two people interrupt each other, a segment belongs to two groups at once. Older methods regularly failed at this. Newer systems are trained from the outset to recognize multiple voices in parallel. Similar-sounding voices, such as two brothers, also remain a typical source of error. Quality is measured using the Diarization Error Rate, i.e., the proportion of time that was incorrectly assigned.

Diarization in meeting tools and podcast apps

The method is most commonly encountered in video conferencing and note-taking tools. Programs like Teams, Zoom, or Otter transcribe meetings and prefix each sentence with a name. In a video conference, the task is easier because each participant has their own microphone. It is more difficult with a room of five people using a single conference microphone.

In practice, many users stumble over a mix-up. Speaker Diarization separates voices but does not recognize identities. If an app actually writes “Anna”, the name comes from the conference account or from a previously stored voice sample. The latter is called speaker recognition and is a separate, privacy-sensitive technology.

In news about AI, the term usually comes up in connection with speech technology companies, such as AssemblyAI, Deepgram, or Speechmatics. They sell transcription as a service, and speaker attribution is a key selling point in doing so. Podcast platforms also use it to create searchable chapters. Anyone wanting to experiment themselves will find a widely used tool in the free software library pyannote.