Four Hundred Flags: Why Automated Call QA Becomes Shelfware in Three Weeks

The failure mode nobody demos

Ask a vendor in this category what their tool misses and you will get a careful answer about recall. Ask what it flags wrongly and the conversation moves on.

That is the wrong way round. The failure mode of automated call QA is not missing things — it is crying wolf.

A speech-analytics tool that flags four hundred calls a week teaches supervisors, within about three weeks, to ignore it. The flags are not wrong, exactly. They are technically correct and practically useless:

• The agent said “lawyer” — while explaining that the customer could consult one.
• The threatening-language detector fired on a scripted regulatory disclosure.
• The negative sentiment was topic-bleed from a genuinely sad conversation the agent handled beautifully.

Each of those is defensible in isolation. Together they are a queue no human can work. The supervisor cannot check four hundred flags, so they check none — and the tool becomes shelfware that still appears on the licence renewal.

The tool did not fail. It succeeded at the wrong thing, at volume, until nobody was listening any more.

Why the incentives produce this

This is not vendor incompetence. It is a rational response to how the category gets evaluated.

A missed violation is a visible failure. It surfaces in a complaint, an audit finding, a regulator’s letter — and it is traceable directly to the tool that should have caught it. A false positive is an invisible failure. It surfaces as a supervisor’s Tuesday afternoon, and it never appears in a quarterly review.

So the cost of recall is borne by the vendor and the cost of precision is borne by the customer. Every product decision made under those incentives pushes the same way: flag more, explain later, let the floor sort it out.

The floor does not sort it out. The floor stops looking.

What precision actually costs

There is no threshold that fixes this. Tighten the rules and you lose the genuinely ambiguous calls, which are the ones worth reading. Loosen them and you are back at four hundred.

The trade-off feels unavoidable because both halves of it are being decided by the same machine. What breaks it is putting a different kind of reviewer between the detection and the report — and accepting what that costs.

A trained analyst reviews every AI flag before it reaches a customer report. That is the operating model, not a premium tier.

Trained analysts verify flags before they reach your reports, so what supervisors act on is real rather than algorithmic noise.

And the analyst reviews in the language the call was spoken in. A human check performed over a machine translation is not really a check — it inherits every judgement the translation already made about tone, idiom and severity. Reviewing natively is what makes the second opinion independent of the first.

What we changed: making verification structural

A promise that humans review things is worth very little on its own. Three mechanisms make the verification enforced rather than aspirational.

1. A mechanical lint gate before the database load

A linter runs before anything is written, and exits non-zero on:

• a negative agent segment carrying no coaching,
• coaching that landed on a customer line rather than an agent one,
• a brand-risk segment that was not forced negative.

Bad batches do not get flagged for follow-up. They fail. Nothing partially-correct reaches storage to be cleaned up later, because “cleaned up later” is how inconsistency enters a dataset that agents will be scored against.

2. Code re-derives the model’s arithmetic and asserts it matches

The analysis step produces both a judgement and the numbers that follow from it. A language model asked to do both will occasionally return a severity band that does not match its own reasoning, or a blended score that does not follow from its own components.

So the build step recomputes them independently and raises if they disagree — the severity band, the blend, the presence of flag evidence. It is a real guardrail against model drift, and it is the kind of thing that only gets built by a team that has watched a model drift.

3. Anti-drift rules on the analyst

Human reviewers drift too, in a different way: by inventing categories mid-run.

Every flag you use must have a one-line “when to use” written down … otherwise the flag is analyst interpretation, not a rule, and results drift between runs. Do not invent suffixes on the fly.

An analyst who improvises a new severity level in week six has quietly made every prior week incomparable. The rule exists because comparability across weeks is the entire point of a trend.

The shape this produces

Detection stays broad, because detection is cheap and a narrow detector misses exactly the ambiguous cases worth reading. Everything downstream of detection is narrow, because that is where your supervisors’ attention gets spent.

What lands in the report is not what the model found. It is what a person confirmed.

What we do not claim

We do not claim our model is right more often than other models. That is not the differentiator, and a precision figure published by the vendor whose precision is in question is worth roughly what you would expect.

We do not claim the process is fully automatic. It is not — and the honest version is more useful: the heaviest analysis stages involve a person, that person is the reason the output is worth acting on, and the cost of them is priced in rather than pushed onto your floor.

We do not claim zero false positives. We claim that the ones you see were reviewed by somebody whose job was to try to reject them.

Where to start

The question worth asking any vendor in this category, including us, is not “what is your accuracy?” It is: who looked at this flag before I did, and what were they allowed to reject?

If the answer is nobody, then you are the reviewer. Price that in before you compare licence fees.

Book a demo and we will show you what the verification step removes — the flags that get raised and do not survive it.

---

Related: What We Refuse to Score on the fairness gates behind the same pipeline, and The 5% Illusion on the coverage problem this one sits downstream of.