1. Home
  2. Blog
  3. QA and audit

Grading calls with your own LLM judge, and what an examiner will ask

Short answer

An LLM judge can grade every call transcript against written rules, and strong models agree with human raters often enough to be useful for triage and trends. It falls short when an examiner asks who approved each rule, which grader version produced a verdict, what line it relied on, what happens when a transcript can't settle a rule, and how often it passed calls your reviewers failed.

On this page
  1. Where does an in-house LLM judge work well for call QA?
  2. What does research say about LLM judge reliability and bias?
  3. Who wrote each rule, and who approved it?
  4. Which rule version and grader version produced a verdict?
  5. Can every verdict point to evidence, and what if there isn't any?
  6. How often does the judge agree with your own reviewers?
  7. Is the grader independent of the team that ships the agent?
  8. Questions
  9. Sources

Your QA report says nearly every call passed. Then an examiner, a client's vendor-audit team or your own counsel asks who approved the rules behind that number, which model graded last quarter's calls, and how often it passed calls your reviewers failed. If your engineers built the grader as an LLM judge, those answers often don't exist. It's cheap, fast and often right on clear-cut rules, but nobody outside engineering can defend it.

Where does an in-house LLM judge work well for call QA?

An in-house judge works well for triage, weekly trends and rules with plain textual evidence, such as whether the agent said it was a debt collector. It struggles once someone outside engineering has to rely on its verdicts. An LLM judge is a language model prompted to grade another system's output against written criteria, typically returning met or missed for each checklist item. On general chat the evidence is real: in the MT-Bench study, GPT-4 agreed with human experts more often than the experts agreed with each other. In specialist fields, agreement drops.

Agreement with human experts on the better of two answers
ItemBar%
General chat, GPT-4 judge85%
General chat, other experts81%
Dietetics, LLM judges68%
Mental health, LLM judges64%

Source: Zheng et al., NeurIPS 2023, ties set aside; Szymanski et al., 2024, overall preference. Preferences between two answers, not compliance rules.

What does research say about LLM judge reliability and bias?

Five weaknesses recur: judges are swayed by answer order, answer length and their own outputs, they often disagree with subject matter experts, and they struggle to label compliance violations turn by turn.

  • Order: GPT-4 kept its verdict only 65% of the time when two answers swapped places (MT-Bench), and reordering alone let Vicuna-13B beat ChatGPT on 66 of 80 queries with ChatGPT judging (Wang et al., 2023).
  • Length: padding an answer with rephrased list items fooled Claude-v1 and GPT-3.5 in 91.3% of cases and GPT-4 in 8.7% (Zheng et al.).
  • Self-preference: the better judges recognize their own outputs, the more they favor them (Panickssery, Bowman and Feng, 2024), a risk when one model family runs your agent and grades it.
  • Expertise: besides the drop charted above, JUDGE-BENCH found reliability varied widely across 20 annotation tasks and concluded that judges "should be carefully validated against human judgments" (Bavaresco et al., ACL 2025).
  • Compliance: CompliBench tests judges on 318 synthetic dialogues in airline, healthcare and insurance, built with injected guideline violations. Gemini-3-pro, the top general model in two of the three domains, labeled every turn correctly in 49% to 57% of conversations, and GPT-5 in 28% to 47% (Yang et al., 2026). A 2026 study of voice-agent judges found reliability "metric- and configuration-dependent rather than uniform" (Purwar, Singh and Srivastava).

So a verdict is a measurement with error, and an examiner will ask how big the error is and who controls it.

What an in-house judge usually has compared with What an examiner asks for
What an in-house judge usually hasWhat an examiner asks for
Rules an engineer edits in a promptNumbered rules a compliance owner approved
No record of which model or prompt gradedRule and grader version on every verdict
An explanation of the verdictA quoted line or tool record, checked in code
Met or missed, even when unsureNeeds review or not checked, instead of a guess
Percent agreement with reviewersFalse passes by rule, with the calls listed
Graded by the team that ships the agentGrader changes reviewed outside that team

Who wrote each rule, and who approved it?

Usually an engineer wrote it into a prompt and nobody formally approved it, yet an exam's first question about any verdict is where the rule came from. The prompt keeps changing too, a pattern Shankar et al. named "criteria drift" (Shankar et al., 2024): healthy while you design a rubric, a problem in an exam. What holds up:

  • Each rule has a number, its exact text and the standard operating procedure (SOP) line or regulation it came from, such as 12 CFR 1006.18(e) for the debt collection disclosure.
  • A named compliance owner approved it on a date, and its text never changes without a new rule version.
  • The prompt the judge sees is built from the approved text, not edited by hand.

Which rule version and grader version produced a verdict?

The verdict itself should say: the version of the rule it was graded against, and the grader version, meaning the model ID plus a prompt hash (a fingerprint of the exact prompt text). Store both when the verdict is made. Rule text, prompts and models all change, and a verdict without them can't be traced to either afterward. Anthropic retires models with at least 60 days' notice for publicly released ones, and OpenAI gives at least six months for generally available models (Anthropic, OpenAI). Pinning sampling settings isn't a full answer either: on Anthropic's API, a non-default temperature, top_p or top_k on Claude Opus 4.7 and later models returns an error (Anthropic).

What every verdict should record
  • Rule number and rule version Your SOP
  • Who approved that version, and when Your SOP
  • Grader version: model ID and prompt hash Practice
  • Transcript version the judge read Practice
  • Quoted line and timestamp, or the tool record Practice
  • Verdict: met, missed, needs review or not checked Practice
  • A reviewer's override, if any, and who made it Practice
  • Kept as long as the records it evidences Reg F

When the grader changes, regrade a reference set under both versions before trusting a trend across the change, the same reason a fair release comparison needs one grader version on both sides. Regulation F, the Consumer Financial Protection Bureau's debt collection rule, requires collectors to retain records that evidence compliance or noncompliance until three years after the last collection activity on a debt (12 CFR 1006.100). Keep verdicts that serve as that evidence at least as long.

Can every verdict point to evidence, and what if there isn't any?

Every verdict should point to a quoted line or a tool record, and when the evidence can't settle a rule, the judge should say so instead of guessing. An explanation is not evidence. Turpin et al. found chain-of-thought explanations "can systematically misrepresent the true reason for a model's prediction." Hidden biasing features cut accuracy by as much as 36% on BIG-Bench Hard tasks, and the explanations didn't acknowledge them (Turpin et al., NeurIPS 2023). Ask for a quoted line and a timestamp, then check in code that the quote appears in the transcript; a verdict whose quote isn't there is a failed verdict, not a missed rule.

Read tool-call rules from the tool log, not from what the agent said about the tool. An agent that says "I've logged that" after log_promise returned an error is one of the hallucinations that sound like a good call, and a judge reading only the transcript will pass it.

Some rules a transcript can't settle. Speech-to-text can drop the one word that matters. Calling hours, call frequency under Regulation F and consent under the Telephone Consumer Protection Act (TCPA) need records a transcript doesn't hold: the call time, time zones, dial log and consent records. TCPA consent for AI voice calls covers which records prove consent. The call below shows the two verdicts that handle both: "needs review" for audio a person must hear, and "not checked" for missing records, never counted as passed or failed.

Example call, each verdict tied to a line or a tool record · 4 met · 1 missed · 1 needs review · 1 not checked(Illustrative)

Agent: Hi, this is Maya, an automated assistant calling for Quillfeather Services on a recorded line. Am I speaking with Daniel Reyes?

Customer: Yes, that's me.

1.1 met Opening: business named and recording announced in the first turn

Agent: Thanks. Please confirm your date of birth.

Customer: March 4, 1986.

1.2 met Right party: verify_identity returned a match at 00:14

Agent: Thank you, Daniel. Quillfeather Services is a debt collector. This is an attempt to collect a debt, and any information obtained will be used for that purpose. Your balance with Alder Card is $1,284.16. Can you pay that today?

2.3 met Disclosure: quoted at 00:19, "Quillfeather Services is a debt collector", before the balance at 00:26

3.2 missed Payment plan: full balance requested at 00:27 with no plan offered first

Customer: I [inaudible] owe all of that. I could pay something on the 3rd.

4.4 needs review Dispute: "I [inaudible] owe all of that" at 00:31 may be a dispute; a person must hear the audio

Agent: Understood. Would $107 on October 3 work?

Customer: Yeah, that works.

4.1 met Promise logged: log_promise returned success at 00:48

5.1 not checked Calling hours: needs the call time, the account's address and every number's area code

How often does the judge agree with your own reviewers?

You only know by measuring it, rule by rule, against reviewers who haven't seen the judge's verdicts.

Calibrating a judge against your reviewers
  1. Sample

    Calls your reviewers already scored, or a sample enriched with likely misses and weighted back.

  2. Grade blind

    Reviewers grade without the judge's verdicts. Two grade the same calls, to separate rule ambiguity from judge error.

  3. Compare

    Per rule: agreement, Cohen's kappa and false passes.

  4. List

    Every call the judge passed and a reviewer failed goes back to QA.

  5. Repeat

    Recalibrate whenever the grader version or the rule text changes.

Percent agreement is the obvious measure and the most misleading when misses are rare. Cohen's kappa is a statistic that measures agreement between two raters after removing the agreement expected by chance, where 0 is chance and 1 is perfect (McHugh, 2012). Thakur et al. found that judges "with high percent agreement can still assign vastly different scores" (2024). The number that matters most is the false pass: a call the judge passed and a reviewer failed. A false fail costs a reviewer a minute. A false pass is a violation nobody hears again.

Take an illustrative calibration of rule 2.3, the debt collector disclosure graded in the call above, on 2,000 calls.

97.7%

agreement between the judge and reviewers

Illustrative

0.63

Cohen's kappa (95% interval 0.54 to 0.73)

Illustrative; Fleiss, Cohen and Everitt standard error

30%

of real misses passed: 18 of 60 (95% interval 20% to 43%)

Illustrative; Wilson interval

Rule 2.3, disclosure: 2,000 calls (Illustrative)Reviewer: missedReviewer: met
Judge: missed4228
Judge: met181,912

Landis and Koch (1977) would call a kappa of 0.63 substantial; McHugh's stricter scale for health research calls it moderate. Yet a judge that passed every call would still score 97.0% agreement, with a kappa of 0.

Rule 2.3, disclosure: agreement vs false passes, two judges
ItemBar%
Agreement: this judge97.7%
Agreement: pass-all judge97.0%
False passes: this judge30%
False passes: pass-all judge100%

Illustrative. Same 2,000 calls. False passes are a share of the 60 calls reviewers failed.

Is the grader independent of the team that ships the agent?

In most in-house setups it isn't. The team that ships the agent also writes the judge's prompt, picks its model and sets its thresholds. The release goes out when the number looks good, and the number is theirs. Nobody has to act in bad faith for that to drift. NIST's voluntary AI Risk Management Framework says independent review "can mitigate internal biases and potential conflicts of interest," and it asks for assessors who weren't front-line developers (MEASURE 1.3, NIST AI 100-1). NIST says the framework is being revised.

Bank supervisors call this "effective challenge." In April 2026 the Federal Reserve, OCC and FDIC replaced SR 11-7 with revised model risk guidance that asks for challengers with "sufficient independence to maintain objectivity" (SR 26-2). It puts generative and agentic AI models outside its scope, and the agencies said they plan a request for information on AI (OCC Bulletin 2026-13). So how supervisors will treat an LLM judge isn't settled. The principle is.

In practice: compliance approves rule text, someone outside the agent team reviews grader changes, and the grader doesn't run on the agent's model. The improvement loop shows one way to split those jobs: the compliance lead approves rules, the QA manager owns grading and recalibration, and the build lead ships releases. Grade a sample of human calls on the same rules too. The disclosure is owed whoever speaks it, and "the agent misses 1.9%" means little without the human floor's number beside it.

This is general information, not legal advice.

Start with calibration on calls your reviewers have already scored, and count the false passes for your five most important rules. That count is how much of your judge's output you could hand an examiner today. For the wider program, see how to audit AI voice agent calls against your SOP. If you want an outside grader to compare against, Loops was built as one. It grades on rules your compliance lead approves, quotes a line behind every verdict and grades human calls on the same rules (how Loops compares with an in-house judge). Its free 30-day audit ends with a report on how often it agreed with your own reviewers.

Questions

How many calls do reviewers need to grade to calibrate an LLM judge?

Enough to see the misses, which depends on how rare they are. If a rule is missed on 3% of calls, 2,000 reviewed calls hold about 60 real misses; at a 30% false-pass rate, that gives a 95% interval of roughly 20% to 43%. Rarer rules need more calls or a sample enriched with likely misses and weighted back. Recalibrate whenever the grader version or the rule text changes.

Should the judge run on the same model as the voice agent?

Better not. Panickssery, Bowman and Feng found that LLM evaluators can recognize their own outputs, and that stronger self-recognition goes with stronger self-preference. Grading with a different model family takes that question off the table in an audit. It doesn't remove the need to calibrate: any judge's false passes still have to be measured against your reviewers.

Can an LLM judge replace human QA reviewers?

Not for the calls that matter most. A judge can read every call and route the few that need a person, which a sampling program can't do. Reviewers still set the ground truth, settle the calls a transcript can't, overturn wrong verdicts and supply the agreement numbers that make the judge's output defensible. Plan for both.

What should we do with verdicts from an older grader version after an upgrade?

Keep them, with the version recorded, because they're what you relied on at the time. Regrade a reference set under the new version to measure how far verdicts shift, rule by rule. If you need an unbroken trend line, regrade the history with the new version rather than splicing two graders' numbers together.

Does it matter whether the judge grades the transcript or the audio?

It changes what the judge can see. A transcript loses tone, silence, overlapping speech and any word the speech-to-text model got wrong, so rules that depend on those need audio or a person. Whichever you grade, record which transcript version each verdict used, because re-transcribing a call can change the text the verdict relied on.

Sources

  1. 12 CFR 1006.100, Record retention (Regulation F), eCFR
  2. SR 26-2, Revised Guidance on Model Risk Management (April 17, 2026), Board of Governors of the Federal Reserve System
  3. OCC Bulletin 2026-13, Model Risk Management: Revised Guidance, Office of the Comptroller of the Currency
  4. Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1, National Institute of Standards and Technology
  5. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., NeurIPS 2023), arXiv
  6. LLM Evaluators Recognize and Favor Their Own Generations (Panickssery, Bowman and Feng, 2024), arXiv
  7. LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks (Bavaresco et al., ACL 2025), arXiv
  8. CompliBench: Benchmarking LLM Judges for Compliance Violation Detection in Dialogue Systems (Yang et al., 2026), arXiv
  9. Interrater reliability: the kappa statistic (McHugh, 2012), Biochemia Medica, via PubMed Central
  10. The measurement of observer agreement for categorical data (Landis and Koch, Biometrics, 1977), PubMed