Why voice bots fail at Hinglish

Hinglish is not two languages taking turns

There are two different things people call bilingual speech, and voice stacks handle exactly one of them well. Inter-sentential switching means finishing a sentence in one language and starting the next one in another. Intra-sentential switching means changing language inside a single clause — and that is what Hinglish overwhelmingly is.

"Sir aapka order dispatch ho gaya hai, tracking link bhej dun?" puts English nouns inside Hindi grammar. The structure is Hindi; the vocabulary is both. There is no point in that sentence that is a clean place to switch a recogniser over.

This is why a language dropdown is the wrong shape for the problem. Choosing "Hindi" at the start of a call tells the system nothing useful about a sentence that is about to be half English nouns, and choosing "English" is worse.

Where the transcript breaks

The root cause is structural and fairly boring: there is not much code-switched training data. A systematic review of end-to-end code-switching speech recognition notes that because large-scale code-switched training data is limited, researchers commonly model each language separately from monolingual data and bring code-switched data in later through transfer learning.

That leaves two weaknesses you can hear on a call.

• Language identity gets thrown away. When a single tokeniser is trained across languages, the per-token language information is discarded unless something external is added to preserve it — so the model is not reliably tracking which language the current word is in.
• Synthetic mixing is not real mixing. Where code-switched data is manufactured by stitching monolingual sources together, words from different languages end up adjacent in combinations that rarely occur in real speech, and the model learns a pattern callers do not actually produce.

The published numbers are blunt about the cost. The MUCS 2021 challenge at Interspeech released around 600 hours of transcribed Indian-language speech across its subtasks and reported baseline word error rates of 30.73% and 32.45% on the two code-switching tracks, Hindi-English and Bengali-English. Work on the HiACC Hinglish corpus cites evaluations showing a relative word error rate increase of 30-50% on code-switched speech compared with monolingual input.

The benchmark is lying to you, in both directions

Here is the part almost nobody selling a voice agent will raise: for romanised Hindi, the standard accuracy metric is itself unreliable.

One Hindi phrase has many legitimate Roman spellings, varying with the writer's habits, their ear, and how fast they type. Word error rate compares a transcript against one reference string, so every legitimate variant is scored as an error — the system is marked wrong for being spelled differently from the one transcription someone happened to write down.

Recent work on orthographically informed evaluation for Indian languages argues exactly this: that strict word error rate paints a harsher picture than a human listener would, and proposes a variant-aware metric that folds spelling and word-formation differences into the score. The same work notes that reaching for character error rate instead overestimates performance, because a single wrong character can still force a user to retype a whole word.

So the number misleads in both directions. Strict WER makes a competent Hinglish system look worse than it is; character error rate makes a poor one look better. Either way, a vendor quoting you a Hinglish accuracy figure has chosen their own denominator.

• Ask what the reference transcription policy was — a single reference string, or multiple accepted spellings?
• Ask whether the test set is real code-switched calls, or monolingual audio concatenated together.
• Ask for the error rate on your own recordings rather than on their benchmark.

The reply fails differently from the transcript

Transcription is the half that gets measured. The reply is the half callers actually notice.

• Grammar that does not agree with the voice. A reply assembled in English word order and spoken in Hindi is intelligible and immediately wrong, in the way a badly translated instruction manual is wrong.
• An English phoneme inventory on Hindi words. A synthesiser trained mainly on English approximates retroflex consonants and aspiration, and the result sounds like a non-native speaker — which a caller reads as an offshore call centre rather than as a computer.
• Register mismatch. Answering a Hinglish question in formal, heavily Sanskritised Hindi is a tonal error. It is grammatical, and it is still the wrong reply, because it sounds like a different person from the one the caller was just speaking to.

People forgive a misheard word — they have been doing that on phone calls their whole lives. What they do not forgive is a voice that sounds wrong, because that is the cue they use to decide whether the thing on the other end is competent.

The latency tax of switching

If a stack handles multiple languages by detecting the language and routing to a per-language pipeline, that detection has to happen before anything else can — and on code-switched speech it has to happen again and again, inside a single utterance.

That cost lands on top of the round trip every voice agent already pays. On a phone call, about a second of silence is the point at which a caller assumes the line has dropped and says "hello?", which the agent then hears as a new turn. A language-detection hop is an expensive place to spend that budget.

How to test this in ten minutes

You do not need a lab. Put the vendor on a real phone call, over a mobile network, and do these six things.

1. Switch language mid-sentence with no pause: "Mujhe apna plan upgrade karna hai, but pehle current usage bata dijiye." Listen for whether the transcript survives the join.
2. Say a long alphanumeric in the middle of a Hindi sentence — an order number, a vehicle registration. Mixed-script strings are where accuracy falls off a cliff.
3. Use an English proper noun with Hindi case marking. Brand names and place names inside Hindi grammar are a common failure.
4. Interrupt the agent mid-reply, in the other language. Barge-in and code-switching together is much harder than either on its own.
5. Ask the same question twice, once in Hindi and once in Hinglish. You should get the same answer in two registers, not two different answers.
6. Ask for a transfer to a human, and look at what that person receives. If the handoff loses the Hinglish transcript, the agent has made their job harder, not easier.

Where this still breaks, including for us

An honest account of the state of the art, ours included:

• Long alphanumeric strings inside mixed speech. Digits spoken in one language with a letter sequence in the other are still the most common reason a confirmation has to be repeated.
• A third language arriving unannounced. A caller who moves from Hinglish into Marathi mid-call is outside what a two-language setup anticipates.
• Heavy regional accent combined with code-switching. Each is manageable; together they compound, and on a poor line they compound further.
• Proper nouns that exist in both languages with different pronunciations — which is most of an Indian address.
• Sentiment in mixed speech. Sarcasm and politeness markers carry in Hinglish in ways a model tuned on English conversation does not reliably pick up.

None of this is an argument against running an agent on Hinglish calls. It is an argument for designing the handoff deliberately, for watching repeat rates rather than containment alone, and for testing on your own recordings before believing anyone's accuracy claim — ours included.

Where these numbers come from

See it on OwnPBX

• Hindi, Hinglish and 10+ Indian languages
• What is an AI voice agent?
• Talk to an agent and try switching mid-sentence
• How the OwnPBX agent works end to end