How to audit AI voice agent calls against your own SOP
Short answer
To audit AI voice agent calls, split your standard operating procedure into numbered, testable rules that your compliance lead approves, then grade every call against each rule. Every verdict should carry the quoted transcript line, the tool-call record, the rule version and the grader version. Send calls a transcript can't settle to a person, calibrate against your own reviewers, and compare miss rates between releases.
On this page
- How do you turn an SOP into rules you can grade?
- What can you grade from the call, and what needs outside records?
- What evidence should each verdict carry?
- What happens when a transcript can't settle a rule?
- Should you sample calls or grade every one?
- How do you calibrate the grader against your reviewers?
- How do you compare one agent release with the last?
- What should you hand an examiner or a client auditor?
- Questions
- Sources
Your AI voice agent handles thousands of calls a week, and the platform's dashboard reports minutes and containment (the share of calls the agent finished without a person). It doesn't show whether the agent made up a balance, skipped a disclosure, promised a payment your system never logged, or started missing a rule after last week's release. An audit answers that per call and per rule: did the agent do what your standard operating procedure (SOP) requires, and can someone outside your team check the answer? A QA scorecard that says a call "scored 87" can't be checked. A verdict that says rule 2.3 was missed, quotes the line and names the rule version can.
This guide covers the audit itself. Who owns each station and how often it runs is in the improvement loop for AI voice agents.
How do you turn an SOP into rules you can grade?
Most SOPs were written to train people, so they mix policy, tone and regulation in one paragraph. "Be transparent with the consumer" can't be graded. "2.3: Before the debt is discussed, the agent says the call is from a debt collector attempting to collect a debt" can.
Rule 2.3 comes from Regulation F (12 CFR part 1006), the Consumer Financial Protection Bureau's rule implementing the Fair Debt Collection Practices Act (FDCPA). 12 CFR 1006.18(e) requires that disclosure in the initial communication. Every later communication needs a shorter one: that it's from a debt collector.
- Split
One behavior per rule: "verify the consumer and explain the call" is two rules
- Trigger
When it applies: a balance comes up, a promise is made, a lawyer is mentioned
- Evidence
A quoted line for what's said, a tool-call record for what the system did
- Cite
The SOP section and, where there is one, the regulation behind it
- Approve
Your compliance lead signs each rule beside the SOP text it came from
- Version
Every later change gets a new number; past verdicts keep the one they were graded on
Here's how a few clauses become rules in an example collections SOP (rev 7). It's fictional, and every guide on this blog uses the same one, so a rule number means the same thing everywhere. FDCPA and Regulation F rules for AI collection calls covers the rest of a collections SOP.
| SOP clause (example) | Rule | Evidence | Settled by the call alone? |
|---|---|---|---|
| "Confirm it's the consumer before discussing the account" | 1.2 Identity verified before any account detail (12 CFR 1006.6(d)) | Verification tool match, before the first account detail | Yes |
| "Give the collection disclosure at the start" | 2.3 Disclosure before the debt is discussed | Quoted disclosure, timestamped before the first mention of the debt | Yes |
| "Quote only the balance on file" | 2.4 Stated balance matches what get_balance returned on the call | Quoted amount against the tool's result | Yes |
| "Log every promise to pay" | 4.1 Promise logged, and the log succeeded | Tool call, its arguments and a success status | Yes |
| "Stop calling when asked" | 4.2 Request acknowledged, collection talk stops, request logged to the dialer (12 CFR 1006.14(h)) | Quoted request and the dialer update | The request, yes; proof of no later calls needs the dial log |
| "Call only 8 a.m. to 9 p.m. the consumer's time" | 5.1 Call placed inside that window everywhere the consumer may be (12 CFR 1006.6(b)(1)) | Call time, the account's address and every number's area code | No, needs account data |
| "No more than seven calls in seven days" | 5.2 Call frequency within Reg F's presumption, per person per debt (12 CFR 1006.14(b)(2)) | Dial log for the person and the debt | No, needs the dial log |
What can you grade from the call, and what needs outside records?
The transcript and the tool-call log settle anything the agent said or did on the call. Rules that span calls, or turn on facts the call never mentions, need the records that hold those facts.
Call frequency is the clearest case. Reg F presumes a collector complies if it calls one person about one debt no more than seven times in seven consecutive days. The second condition: no calls in the seven days after a phone conversation about that debt. Only the dial log can show either.
Consent needs consent records. The Telephone Consumer Protection Act (TCPA) is the federal law that restricts calls made with an artificial or prerecorded voice. Under 47 U.S.C. 227(b)(1), those calls need prior express consent when they go to a cell phone, and often when they go to a residential line. The FCC's February 2024 declaratory ruling, FCC 24-17, says AI-generated voices count as artificial. Since the Supreme Court's June 2025 decision in McLaughlin Chiropractic v. McKesson, district courts interpret the TCPA for themselves rather than deferring to the FCC. Read how TCPA consent applies to AI voice calls before relying on any single reading.
Without those records the verdict is "not checked": never passed, never failed. An audit that quietly passes cross-call rules reports compliance it never tested.
Healthcare and insurance split the same way. Identity verification before disclosing protected health information (45 CFR 164.514(h)) shows up in the call. A restriction the patient asked for last month lives in the patient record. See HIPAA rules for AI voice agents in patient access and AI voice agents on insurance claims calls.
What evidence should each verdict carry?
- The quoted line and its timestamp, or the stretch where the behavior should have appeared Practice
- The tool-call record: which tool, its arguments, what it returned and when Practice
- The rule version and who approved it Your SOP
- The grader version: the model, prompt and settings that produced the verdict Practice
- Any reviewer override, with who made it and when Practice
The tool-call record is where invented promises and other hallucinations surface. An agent that says "you're all set" while log_promise returned an error has made a promise your system doesn't know about.
Keep verdicts as long as the law keeps the calls. For collections, a verdict is a record that evidences compliance or noncompliance. Reg F says to keep those from the start of collection on a debt until three years after the last collection activity on it (12 CFR 1006.100(a)). That usually outlasts the three years a call recording must be kept from its call date (1006.100(b)). Treat verdicts as material an examiner or a plaintiff can ask for, and settle with counsel how misses are reviewed and remediated.
Agent: Hi, this is Maya, an automated assistant calling for Quillfeather Services on a recorded line. Am I speaking with Priya Shah?
1.1 met Opening: business named and recording announced in the opening
Customer: Yes.
Agent: Thanks. To confirm it's you, what's your date of birth?
Customer: June 2, 1979.
1.2 met Right party: verify_identity returned a match at 00:17, before any account detail
Agent: Thank you. Quillfeather Services is a debt collector. This is an attempt to collect a debt, and any information obtained will be used for that purpose. Your Alder Card balance is $2,310.40.
2.3 met Disclosure: given at 00:21, before the balance at 00:29
2.4 met Balance: matches get_balance, returned at 00:26
Customer: I can't pay that. [inaudible] stop calling [inaudible]
Agent: I understand. Could you manage $150 on the 15th?
4.2 needs review Stop calling: possible request is partly inaudible; listen to 00:34 to 00:38
Customer: Okay, the 15th.
Agent: You're set for $150 on October 15.
4.1 missed Promise logged: log_promise returned an error at 01:05 and was not retried
5.2 not checked Call frequency: needs the dial log for Priya and this debt
What happens when a transcript can't settle a rule?
The call goes to a person, marked for review with the question to answer and the timestamps to hear. Transcripts fail in ordinary ways: people talk over each other, tool logs go missing, and consumers say things like "I think my lawyer has this." Speech recognition also errs more for some callers than others. In a 2020 Stanford study, five commercial systems averaged a word error rate of 0.35 for Black speakers and 0.19 for white speakers (Koenecke et al., PNAS).
A grader forced to pick met or missed will guess, and its guesses look exactly like findings. Language models lean that way: a 2025 paper traced their hallucinations to training and evaluation that "reward guessing over acknowledging uncertainty" (Kalai et al.). So give the grader a third answer, needs review, and let it use it.
Track the review rate per rule, one of the KPIs that show whether an agent follows your rules. When one rule sends many calls to review, check its wording and whether the grader gets the data it needs. Fixing the rule once beats reviewing those calls forever.
Should you sample calls or grade every one?
Grade every call, then sample the verdicts.
Under 5%
of conversations is often all that manual QA assesses, McKinsey estimated
About 10%
chance a 200-call sample catches even one of 5 third-party disclosures in 10,000 calls
Illustrative
On those illustrative 10,000 calls, the same sample holds about 2 misses on a rule broken 1 percent of the time. A release that doubled that rate would show 4, which one week can't tell apart from noise.
Reviewers then stop choosing which calls to hear and start checking the grader. They read every miss on a severe rule, every call marked for review, and a random slice of passes. The passes matter most. A grader that wrongly passes calls fails silently, and only a sample of its passes will show it.
How do you calibrate the grader against your reviewers?
Measure how often the grader agrees with your own reviewers, on your own calls, rule by rule.
- Sample
A few hundred of your calls, stratified so every rule has misses in it
- Baseline
Two reviewers grade the set independently; measure their agreement first
- Grade
Run the grader over the same calls, rule by rule
- Kappa
Report agreement per rule with Cohen's kappa, not raw agreement
- False passes
List every call the grader passed and a reviewer failed; an average hides them
- Rerun
Grade the set again whenever the grader version or a rule changes
Reviewers aren't ground truth either. How often they agree with each other shows how much of the disagreement comes from the rule itself. Report Cohen's kappa, which corrects for agreement expected by chance (McHugh, 2012). Raw agreement flatters rare rules. On a rule missed 2 percent of the time, a grader that passes every call agrees 98 percent of the time and has a kappa of zero. If you built your own grader, see what an examiner will ask about your LLM judge.
How do you compare one agent release with the last?
Tag every call with the agent version and compare each rule's miss rate between versions on live calls. Three things keep the comparison fair.
- Keep the grader version the same on both sides, or you're measuring a grader change. A model can change under an unchanged name: GPT-4 identified prime numbers with 84% accuracy in its March 2023 version and 51% in June (Chen, Zaharia and Zou).
- Match the call mix (same weeks, clients and call types), or a traffic shift reads as a regression.
- Put a 95% interval on each change: the range the true change likely falls in. Newcombe's method, which combines a Wilson score interval for each release's rate, performed well in his 1998 comparison of eleven methods (Newcombe, Statistics in Medicine).
Then attach the calls behind every change, so an engineer reads them instead of arguing about a percentage.
| Rule | Interval | Change and read |
|---|---|---|
| 1.2 Right party | −0.6 pts (−0.9 to −0.3) Improved | |
| 2.3 Disclosure | +1.6 pts (+1.3 to +1.9) Regressed | |
| 3.2 Payment plan | −4.7 pts (−5.8 to −3.6) Improved | |
| 4.1 Promise logged | −3.3 pts (−5.0 to −1.7) Improved | |
| 4.2 Stop calling | +3.2 pts (−0.6 to +7.3) Too close to call |
Illustrative. Same weeks, same grader version on both sides; Newcombe 95% intervals.
In this illustrative release story, v14 improved three rules and broke one. Rule 2.3 went from 0.3% (30 of 10,040 calls) to 1.9% (190 of 10,010), and those 190 calls became the remediation list. Rule 4.2's range crosses zero, so it's too close to call.
Pre-release test calls only cover the paths someone thought to script. Why a release that passes its tests can still break a rule shows how that happens, and how a change in call mix alone can move a total. Loops runs this comparison on every live call, with the same grader version on both sides. Joined to your own payment records, which Loops doesn't read, the same release split shows what each release earned.
What should you hand an examiner or a client auditor?
For any call, account or date range: the verdict evidence above, the calibration report, and a remediation list of accounts that need follow-up.
Examiners look at whether you check. The CFPB's compliance management review procedures (August 2017) ask whether "the institution is determining that transactions and other consumer contacts are handled according to the entity's policies and procedures." They also expect institutions to oversee their service providers' compliance, which is one reason creditor clients audit the agencies that collect for them.
The same procedures separate monitoring from audit. Monitoring may be done by the business unit itself. Audit is independent of both the compliance program and the business functions, and it reports to the board or a board committee. So a grader your own team runs is monitoring, whoever built it. It supports audit when internal audit, or an outside party independent of compliance and the business, uses it and reports to the board.
CFPB supervision reaches nonbank collectors with more than $10 million a year in receipts from consumer debt collection (12 CFR 1090.105(b)). A smaller agency hears the same questions from the creditors it collects for.
Grade your human agents on the same rules. Transcribe floor recordings and score them the same way, using account notes and disposition codes where a person has no tool-call record. One rubric answers a question any client auditor can fairly ask: is the AI agent more or less compliant than the people beside it?
This is general information, not legal advice.
Start with one SOP section this week: split it into numbered rules, get your compliance lead's approval, grade last month's calls against them, and read every miss before you trust a rate. To see it on your own calls first, Loops' free 30-day audit grades last month's calls against your approved rules.
Questions
What is the difference between monitoring and auditing AI agent calls?
In the CFPB's compliance management review procedures, monitoring is more frequent and less formal and may be carried out by the business unit, while audit is less frequent, more formal, independent of both the compliance program and the business functions, and reports to the board or a board committee. A grader your compliance team runs is monitoring, whoever built it. It supports audit when internal audit or an outside party independent of compliance and the business uses it and reports to the board.
How long should a collection agency keep call recordings and audit records?
Regulation F requires a debt collector to keep records that evidence compliance or noncompliance from the start of collection activity on a debt until three years after the last collection activity, and each recorded call for three years after the call (12 CFR 1006.100). Verdicts, quotes, rule versions and overrides are that kind of record, so they follow the longer schedule, which usually outlasts the recording. Client contracts or state law can require longer.
Do you need call audio, or is the transcript enough to audit a call?
For most rules, the transcript and the tool-call log are enough: they show what was said, in what order, and what the agent's tools returned. Audio matters when the transcript is unreliable, such as garbled numbers, crosstalk or an inaudible stretch, which is exactly when a call should be marked for review so a person can listen to the flagged seconds.
When should audit rules be re-approved or given a new version?
Whenever the SOP changes, a regulation or client requirement changes, or a rule's review rate shows it is ambiguous. Give each change a new version number, record who approved it, and grade new calls on the new version without silently re-grading old ones, so every past verdict still points to the exact rule it was judged against.
Sources
- 12 CFR 1006.18, False, deceptive, or misleading representations or means (Regulation F), eCFR, National Archives
- 12 CFR 1006.14, Harassing, oppressive, or abusive conduct (Regulation F), eCFR, National Archives
- 12 CFR 1006.6, Communications in connection with debt collection (Regulation F), eCFR, National Archives
- 12 CFR 1006.100, Record retention (Regulation F), eCFR, National Archives
- 47 U.S. Code 227, Restrictions on use of telephone equipment, Legal Information Institute, Cornell Law School
- Declaratory Ruling on AI-generated voices under the TCPA (FCC 24-17), Federal Communications Commission
- McLaughlin Chiropractic Associates, Inc. v. McKesson Corp., No. 23-1226 (June 20, 2025), Supreme Court of the United States
- Compliance management review examination procedures (August 2017), Consumer Financial Protection Bureau
- 12 CFR 1090.105, Consumer debt collection market (larger participants), eCFR, National Archives
- AI mastery in customer care: Raising the bar for quality assurance (July 31, 2024), McKinsey & Company
- Interrater reliability: the kappa statistic (McHugh, 2012), Biochemia Medica, via PubMed Central
- Racial disparities in automated speech recognition (Koenecke et al., 2020), Proceedings of the National Academy of Sciences, via PubMed Central
- Why Language Models Hallucinate (Kalai, Nachum, Vempala and Zhang, 2025), arXiv
- How is ChatGPT's behavior changing over time? (Chen, Zaharia and Zou), arXiv
- Interval estimation for the difference between independent proportions: comparison of eleven methods (Newcombe, 1998), Statistics in Medicine, via PubMed