Intonation

When people speak, the voice goes up and down. Some syllables become louder, others are drawn out, and pauses fall in between. This rising and falling is called intonation. The sentence “You’re coming along” can, through this, become a statement, a question, or a command – the words stay the same, only the melody changes. Intonation therefore carries part of the meaning that isn’t visible at all in the written form. This is precisely why it is a central problem for computers that are supposed to speak or listen.

Why artificial voices without melody sound unpleasant

Early text-to-speech computers sounded tinny and monotonous. The reason was rarely the pronunciation of individual sounds, but the missing melody. A sentence without a stress contour comes across to listeners as indifferent, sometimes even unfriendly. People recognize such voices as artificial immediately.

Conversely, the following holds: as soon as the intonation is right, the ear forgives remarkably many small errors. That’s why developers don’t judge the quality of speech systems solely on whether every word is correct. They have test subjects rate how natural a voice sounds. This rating is called the Mean Opinion Score, or MOS for short, and usually ranges from 1 to 5. Modern systems come very close to real human recordings on this scale.

Intonation is also a matter of intelligibility. Wherever a speaker places a pause, they structure the sentence. “Let’s eat, Grandpa” and “Let’s eat Grandpa” differ audibly only through this pause. A text-to-speech system that misjudges the structure of a sentence produces sentences that are simply hard to follow.

What speech melody is made up of

Technically, intonation is described through three measurable quantities. The first is the fundamental frequency, i.e. how fast the vocal folds vibrate. It determines whether the voice sounds high or low, and is measured in hertz. The second quantity is loudness, and the third is the duration of individual sounds and pauses. Experts group these three under the term prosody.

Older speech synthesis worked with fixed rules: at the end of a question, the pitch rises; before a comma there’s a short pause. Such rules work, but sound schematic, because real speech knows far more variations. Today’s systems instead learn the melody from large amounts of recorded speech. The model sees text along with the corresponding real recording and derives on its own which melody fits which sentence.

One problem remains: there is no single correct melody for a given sentence. The same text can be spoken in a friendly, annoyed, or bored manner, and all variants are correct. Models that simply learn the average of all training examples therefore end up again with a flat, bland melody. Better systems are given additional control information, such as a desired mood or a sample recording whose speaking style they are meant to imitate.

Intonation in assistants, audiobooks, and fraud cases

Most often, one encounters this topic with voice assistants and navigation devices. Read-aloud functions in browsers and phones also use speech synthesis. A growing market is automatically generated audiobooks and podcast voices. Whether such a product is any good is decided almost always by the intonation, not the pronunciation.

The reverse direction is equally important. Systems meant to understand spoken language also evaluate the melody. Call center software uses this to assess whether a caller is upset. Distinguishing between a question and a statement, too, often succeeds only via the pitch at the end of the sentence.

In the news, intonation also comes up in connection with voice cloning. A few seconds of recording are enough today to reproduce a voice along with its typical speech melody. Fraudsters use this for fake calls, for instance posing as a family member. A common misconception is that such fakes can be recognized by their robotic sound. That has simply not been true for several years now.