What is an AI voice agent?

An AI voice agent is software that holds a spoken phone conversation without a human on the line. It transcribes what the caller says, decides what to do using a language model and your business rules, and replies in a synthesised voice — then either completes the task or hands the call to a person. Unlike an IVR, the caller is not choosing from a menu: they say what they want in their own words.

Karan Sharma · Updated 2026-10-29

It is not an IVR with better audio

An IVR is a decision tree. You press 1, it moves to a branch; you press 9, it reads the menu again. The caller's job is to translate what they want into the options someone wrote down in advance, and the system fails the moment their need was not anticipated.

A voice agent inverts that. The caller says what they want, and the agent works out what to do. There is no menu to mistranscribe into, and no branch to fall off.

	IVR	AI voice agent
Caller input	Keypress, or a single spoken keyword	Natural speech, any phrasing
Handles the unanticipated	No — falls through to an operator	Usually, if it has the information
Changing it	Re-author the call flow	Edit instructions and knowledge
Interruption	Caller waits for the prompt to finish	Caller can cut in mid-sentence
Cost per call	Negligible	Per minute of agent speech

What happens during a call

Three things run in a loop, and the quality of an agent is mostly the quality of the seams between them:

Speech to text. The caller's audio is transcribed continuously, not after they stop talking. The agent needs to know when a sentence has ended — which is harder than it sounds, because people pause mid-thought and finish without any audible signal.
Deciding. A language model reads the transcript so far, along with your instructions, your knowledge base and whatever the system already knows about the caller, and decides what to say or do next. "Do" can mean looking up an order, booking a slot, or transferring the call.
Text to speech. The reply is synthesised and played. Good agents start speaking before the whole reply is generated, because waiting for the full sentence costs a second the caller notices.

Every turn pays all three costs, plus the network time between them. That total is the only number that decides whether an agent feels like a conversation or like a bad phone line.

Latency is the whole product

On a phone call, silence is the signal people read. Humans leave roughly two tenths of a second between turns; past about a second, a caller assumes the line dropped and says "hello?" — which the agent then hears as a new question, and the conversation desynchronises.

This is why where the agent runs matters more than which model it uses. An agent bolted onto a phone system from outside sends audio out to a third-party service and waits for it to come back, and that round trip is added to every single turn. An agent running inside the calling infrastructure skips it.

Hindi, Hinglish and the code-switching problem

Most voice stacks are built English-first and have a Hindi mode. That is not the same as handling how people in India actually speak, which is both languages inside one sentence — "haan main kal call karunga, but time confirm karna padega".

Two things break when a stack treats this as a language-selection problem. The transcription has to switch mid-utterance rather than at the start of a call, and the reply has to come back in the same register, with grammar that agrees with the voice — a Hindi sentence spoken by a model that has only been trained to pronounce English words sounds wrong in a way callers immediately distrust.

Ask any vendor to demo an interruption and a mid-sentence language switch in the same call. It separates stacks that handle code-switching from stacks that handle two languages.

What it is genuinely bad at

Anything where being wrong is expensive and unverifiable — a quoted price, a medical instruction, a commitment on your behalf. Constrain these to looked-up facts or route them to a person.
Long multi-part tasks where the caller changes their mind three times. Agents lose the thread later in a call than they used to, but they still lose it.
Heavy accents on a poor line, where even a human asks for a repeat. The failure mode to design for is a graceful handoff, not a third attempt.
Emotional calls. An agent can detect that someone is angry; it cannot be the thing that calms them down.

None of these are arguments against deploying one. They are arguments for deciding, in advance, what the agent does when it is out of its depth — which is a design decision, not a setting.

How to tell whether it is working

Call volume handled is the wrong metric, because an agent will happily handle a call badly. Four numbers are worth watching:

Metric	What it tells you
Containment rate	Share of calls the agent finished without a human. The headline number, and meaningless on its own.
Handoff reason	Why it gave up. A rising "did not understand" means the knowledge base, not the model, needs work.
Caller repeat rate	How often callers had to say something twice. The earliest signal that transcription or latency is degrading.
Resolution after handoff	Whether the human could pick up where the agent left off. If not, the agent is passing calls without context.

A containment rate of 60% with angry callers is worse than 35% with clean handoffs. Read them together or do not read them.

Where it should sit

An agent that cannot transfer a call, read your extension list, or appear in the same call reports as everyone else is a separate product wearing your phone system's caller ID. The questions worth asking a vendor:

Can it transfer to a specific extension or queue, and does the person who answers see the transcript?
Do its calls appear in the same reporting as human calls, or in a second dashboard?
Whose carrier are the calls on — can it run on numbers and rates you already have?
What happens to a call in progress when the AI service is unavailable?

That last one is the test. A phone system whose calls fail when an AI dependency is down was not built as a phone system.