Two Supervisors, Two Scores: The Subjectivity Nobody Measures

The experiment nobody runs

Take one recording. Give it to two supervisors with the same rubric, in separate rooms. Compare the scores.

Most QA leads have a rough idea of what would happen and a good reason not to find out. The scores diverge — sometimes by a band, often on the specific skills that matter most, and almost always on anything involving tone, empathy or escalation handling.

Run the harder version: give the same call to the same supervisor twice, a month apart, without telling them. The scores still diverge.

This is not incompetence, and it is not a training problem you can fix with a calibration session. It is what happens when a rubric written in natural language is applied by a tired human at the end of a shift, sixty calls into a week, under pressure to finish.

The rubric was never the thing being applied. The supervisor’s reading of the rubric, this afternoon, was.

Why it stays invisible

Two reasons, and both are structural.

Nobody wants to measure it. An inter-rater agreement study on your own QA programme has exactly one possible outcome that is comfortable, and it is not the likely one. So the inconsistency is never quantified, which means it is never managed, which means it compounds quietly for years.

A rubric strict enough to be reproducible is usually too crude to be useful. This is the genuine tension. You can write a rubric so mechanical that two people always agree — and it will score “did the agent say the words” rather than “did the agent do the job”. Most vendors resolve this by writing a rich rubric and hoping, which produces exactly the divergence above.

The consequence is that agent scores are not comparable between supervisors, between teams, or across time. Which quietly destroys the only two things scores exist for: ranking people fairly, and seeing whether anything is improving.

What we changed: split the rubric, do not soften it

The resolution is not a better rubric. It is separating the part that must be reproducible from the part that requires judgement, and refusing to blend them.

The mechanical half

Three call stages, each with a locked set of skills:

opening%  = opening_hit / 3  × 100        # 3 skills, locked
handling% = handling_hit / 6 × 100        # 6 skills, locked
closing%  = closing_hit / 1  × 100        # 1 skill,  locked

Call Score = opening%/100 × W_open
           + handling%/100 × W_hand
           + closing%/100 × W_close

Four properties make this reproducible rather than merely arithmetic:

• The bucket sizes are locked and enforced by assertion, not by convention. The code asserts that the opening bucket holds exactly three skills and that the weights sum to 100. If a configuration change breaks either, the run fails rather than silently producing scores on a different scale than last week.
• Only the weights are configurable per workspace, defaulting to 40/40/20. A collections floor can decide that handling matters more than opening. It cannot decide that opening has four skills this month.
• Skill presence is binary after alias-collapse. Saying “my name is…” five times scores identically to saying it once. This is deliberate: the rubric measures compliance with a script, not enthusiasm for it, and rewarding repetition would teach agents to repeat.
• The rubric is fetched per workspace and never hardcoded — with the configuration source recorded on the run, so a score can always be traced to the rubric that produced it.

The judgemental half

Five perspectives, each graded fine / small / serious, applied by the analyst: customer satisfaction, rule adherence, escalation handling, problem resolution, agent tone.

These require understanding a conversation. No rule set produces them, and pretending otherwise is how you get the crude-but-reproducible rubric that nobody trusts.

And they never blend

The two halves appear side by side in the report and are never merged into a single number.

This is the decision that makes the rest work. A blended score tells you an agent is at 62% and tells you nothing about what to do. Two numbers side by side tell you whether you are looking at a script problem — the agent is not following the process — or a conduct problem — the agent is following it badly. Those have completely different remedies, and merging them destroys the distinction permanently.

The depth underneath

The taxonomy beneath the rubric is not a flat keyword list, and the difference matters for consistency.

600+ behaviour tags roll up into skills, and a skill can require multiple tag groups simultaneously. Acknowledging an objection, for example, fires only when the call contains both an objection and an acknowledgement of it.

That is the difference between asking “did the agent acknowledge an objection that actually happened” and “did the agent say a reassuring phrase at some point”. Most keyword-scoring products ask the second question and report it as the first — which is another source of divergence, because the same phrase means different things depending on what preceded it.

What we do not claim

We do not publish an exact tag count. You will see “600+” here rather than a precise figure, and that is deliberate: the live taxonomy is the source of truth, and a number quoted from an export is a number that can drift from it. We would rather be checkably conservative than precisely wrong.

We do not claim the rubric is the right rubric for your floor. It is a consistent one. Whether opening should be worth 40% on a collections campaign is your decision, and it is configurable for exactly that reason.

We do not claim machine scoring removes judgement. It relocates it. The judgement is still there, in the analyst’s five perspectives — the claim is that it is now sitting next to the mechanical score instead of dissolved into it.

Where to start

Run the experiment. Take one call, give it to two supervisors, compare.

Whatever gap you find is the noise floor under every agent ranking you currently produce — and it is the number worth carrying into any conversation about QA tooling, including one with us.

Book a demo and we will walk you through how the two halves of the score are kept apart, and what each one is for.

---

Related: What We Refuse to Score on the fairness gates that sit alongside this, and A Score Is Not an Instruction on what has to happen after a consistent score exists.