Voice Agent

A voice agent is a program you can talk to as you would with a person on the phone. It picks up the call, listens to the question, replies with an artificially generated voice, and asks follow-up questions of its own. Unlike a recorded announcement with keypad selection, it needs no fixed menus. You simply say what you want, in your own words. The agent can additionally take action: schedule an appointment, take an order, change an address. It is precisely this ability to act that distinguishes it from a pure voice assistant, which merely provides information.

Why companies are suddenly turning to phone AI

Phone calls are expensive for companies. Every call ties up a person for several minutes. In call centers, this creates by far the largest cost blocks. A voice agent can conduct thousands of conversations simultaneously, around the clock, without hold queues.

There’s also a quality argument. Many calls are pure routine: opening hours, package status, rescheduling an appointment. If the machine handles these cases, employees have more time left for difficult concerns. That’s why the field is being closely watched on the stock market. Companies like ElevenLabs or Deepgram are now valued in the billions.

But there is a downside. Call center work is an important economic sector in countries like India or the Philippines. If voice agents take over simple conversations, jobs are lost there. In addition, customers get annoyed when an agent cannot help further and still fails to transfer them to a human.

From sound signal to spoken answer

Classically, a voice agent consists of three building blocks. The first converts spoken language into text; this is called speech recognition. The second building block is a language model, i.e. an AI system that formulates a fitting answer from text. The third building block reads this answer aloud using an artificial voice. This chain runs anew for every sentence.

The hardest task is timing. A conversation only feels natural if the response begins after about half a second. This delay is called latency. The agent must also recognize whether the person has finished speaking or is just pausing briefly to breathe. Misjudgments lead to the typical effect of both sides talking over each other.

Newer systems skip the detour via text. They process sound directly and output sound directly. This is faster and preserves nuances such as hesitation or irony. For an agent to actually get tasks done, it is additionally connected to databases and calendars. Within clearly defined limits, it may then read and write entries itself.

Hotline, drive-in, and dental practice

Voice agents are most commonly encountered on company phone lines. When a voice asks follow-up questions instead of reading out keypad menus, such a system is usually behind it. Doctors' offices and trade businesses also use them to take calls outside office hours. At some fast-food chains in the US, they take orders at the drive-through lane.

A related term is the voice assistant on a phone or speaker. Alexa and Siri originally belong to this older category: short commands, fixed capabilities. Both manufacturers are currently rebuilding them into genuine voice agents capable of holding longer conversations. The boundary is therefore currently blurring.

In the news, voice agents often come up in connection with fraud. The same technology enables fake calls using the cloned voice of a relative. Consumer protection advocates therefore recommend calling back when money is demanded over the phone. Another point of contention is disclosure. In the EU, the AI Act, the European AI law, requires that you be informed when you are speaking with a machine.