Why most call-analytics vendors mishandle Hinglish — and the demo questions that expose it

Issue 4 argued that the language of your call audits matters. This week is the other half of that argument: the technology side, and why the demo you sat through last quarter probably didn’t show you the problem.

Most collections floors in north India run on Hinglish — not Hindi, not English, but live switching between the two, often inside a single sentence. A borrower says “EMI bounce ho gayi, salary abhi credit nahi hui” and no part of that sentence is fully in either language. Every call-analytics vendor will tell you they support it. The technical reality underneath that word “support” varies enormously, and the variation is exactly where compliance risk hides.

None of this requires you to become a speech-technology expert. It requires four questions, and knowing why each one matters.

What Hinglish actually is — and why it doesn’t reduce to either language

Code-mixed speech is not two languages taking turns. The grammar is usually Hindi’s, the vocabulary borrows from both, English verbs get conjugated with Hindi auxiliaries — “call karo,” “payment ho jayega” — and the same word can be doing different work in each language. Add the script problem: when transcripts are produced in Latin characters, one Hindi word can surface with half a dozen spellings, none of them standardised.

A model trained on one language treats the other language’s words as noise to be resolved into something it knows. That single fact drives almost every failure that follows.

Where the technology actually fails

Speech recognition trained on the wrong mix. An English-trained transcription engine hears a Hindi word and does the only thing it can: substitutes the nearest English sound. A Hindi-trained engine does the same in reverse. The errors are not evenly spread — they cluster at exactly the switch points, and on a collections call the switch points are rarely small talk. Amounts, dates, threats, promises and objections are precisely where speakers shift language mid-sentence.

Translation pipelines that launder the tone out. A common architecture transcribes the call, translates everything into English, and runs the analysis on the translation. It looks sensible and it discards the layer a compliance programme most needs. Hindi carries register in ways English translation flattens — the distance between aap and tu is a compliance fact, and it does not survive the journey. A softly-spoken “phir humein aapke ghar aana padega” can arrive in the analytics layer as a neutral statement about a home visit, its menace filed off in transit. Issue 6 named the patterns that don’t sound like threats; a translation pipeline is how those patterns become invisible to the tool that was bought to catch them.

Sentiment and intent models raised on monolingual text. Most were trained on English — much of it not Indian English — and some on formal Hindi. A model of either kind, shown code-mixed collections speech, is being asked about a language it has never seen at volume. It will still return a score. That is the uncomfortable part: nothing in the output looks different when the model is guessing.

The common thread: every failure produces a clean-looking transcript and a confident-looking dashboard. The errors don’t announce themselves. They surface months later, as a complaint the analytics never flagged.

Three questions, three demos

The questions to ask any vendor — including us; The Heartbeat sells call auditing, so treat this list as one to use on our demos too:

1. “What was your speech model trained on, and what happens at a code-switch point?” A serious answer names the language mix and explains the handling. A worrying answer is the word “multilingual” with nothing after it.
2. “Will you measure accuracy on our calls, not your sample set?” Vendor benchmark recordings are clean, scripted and often monolingual. Your floor is none of those things. The only transcription accuracy that matters is the one measured on your own audio.
3. “Does the analysis run on the original code-mixed text, or on a translation?” You now know why this one matters. If the answer is a translation, ask what happens to register and politeness-borne threat on the way through.

And the demos to insist on, in place of the standard walkthrough: a live transcription of a real (consented, anonymised) call from your own floor; the same call run twice, to see whether the scores agree with each other; and a polite-threat example of your choosing — not theirs — to see whether the system flags what Issue 6 described.

The four-question checklist

For the file, before any contract:

1. Which base models produce the transcript, and what language mix were they trained on?
2. What is the transcription accuracy on our calls — measured, not quoted?
3. Does every downstream analysis run on the original code-mixed text or on a translation?
4. Show us, live, a softly-worded Hindi threat being flagged. On our audio, not the sample library.

A vendor who handles Hinglish well will enjoy answering these. The discomfort of a vendor who doesn’t is itself the evaluation.