1. Home
  2. Blog
  3. Measurement

AI voice agent KPIs: the scorecard for collections, claims and patient access

Short answer

Track five AI voice agent KPIs on every line: each rule's miss rate, calls needing review, false passes against your reviewers, transfers by reason and cost per resolved call. Add two or three outcome KPIs for the line: payments per account reached in collections, claims opened on the first call in insurance, visits kept in patient access. Read each per release, with its own denominator and a range.

On this page
  1. Why don't containment and minutes show whether the agent works?
  2. Which KPIs does every AI voice agent need?
  3. Which outcome KPIs fit collections, insurance claims and patient access?
  4. What does a one-release scorecard look like?
  5. Why do calls go to a person, and when is that good?
  6. How should you read these KPIs release by release?
  7. What does a weekly KPI review look like?
  8. Questions
  9. Sources

A voice platform's analytics report what the platform can see. Retell's post-call analysis, for example, returns a call summary, the caller's sentiment, whether the call met success criteria you define, and whether it reached voicemail (Retell). You can add custom fields, but the analysis still reads one call at a time. It never sees your payments file, your claim system or your appointment book, and it can't tell you whether a release made the agent follow your rules better or worse.

That's the job of a KPI set, and the one below has two layers. Five KPIs apply to every AI voice agent, whatever it does. A few outcome KPIs depend on the line it works: collections, insurance claims or healthcare patient access.

Why don't containment and minutes show whether the agent works?

Containment is the percentage of answered calls that ended without a transfer to staff. It climbs when the agent stops handing off disputes, lawyers and callers who ask for a person, which is the opposite of what you want. Minutes climb when the agent is slow. In the sample scorecard below, v14 read 81% contained over 42,310 minutes, and neither number shows that it skipped a required disclosure on 190 calls.

Keep both on the report as volume counts. Steer by the KPIs that know your rules and your outcomes.

Which KPIs does every AI voice agent need?

These five apply to a collector, a claims line and a scheduling line alike. The last column matters most, because most KPI mistakes are denominator mistakes.

KPIThe question it answersCount it per
Miss rate, per ruleDoes the agent follow each rule?Call where the rule applied
Calls needing reviewCan the transcript settle your rules?Graded call
False passesDoes the grader miss what your reviewers catch?Pass your reviewers rechecked
Transfers, by reasonWhere do people still step in, and why?Answered call, split by reason
Cost per resolved callWhat does a finished outcome cost?Call that reached an intended end

Miss rate per rule. Each rule is a numbered line from your standard operating procedure (SOP), and each gets its own rate, counted only over the calls where it applied. A stop-calling rule divided by every call looks tiny; divided by the calls where someone asked to stop, it's the real rate. Never blend rules into one score, which can rise while the one rule carrying legal risk gets worse.

Calls needing review. This is the share of calls where at least one rule couldn't be settled from the transcript: a garbled number, crosstalk, a request half heard. A rising share means a rule is worded ambiguously or the audio got worse.

False passes. These are calls the grader passed on a rule and your reviewer failed, the errors an examiner or a client auditor finds first. Sample passes every week, not only misses, and report agreement per rule as Cohen's kappa, which adjusts raw agreement for what chance alone would produce (McHugh, 2012). Grading calls with your own LLM judge covers the calibration.

Cost per resolved call is the total cost of a set of calls, staff time on transfers and reviews included, divided by the calls that ended where they should. How to measure what a release earned works through the method and shows how far the platform bill alone understates it.

Which outcome KPIs fit collections, insurance claims and patient access?

Pick two or three for the line the agent works, each with its own denominator.

CollectionsInsurance claimsPatient access
The outcomeDollars posted per 10,000 accounts reachedClaims opened on the first call with a claim number and a dated note, per claim reportedVisits kept per 1,000 patients reached
The commitment on the callPromises per right-party contact, and the share keptClaim forms and instructions sent, per claim openedVisits booked per scheduling call, read beside visits kept
The rule with most at stakeDebt collector disclosure before the debtCoverage stated only from this call's policy lookupIdentity and authority verified before any health details
The early warningDisputes per 1,000 right-party contactsRepeat calls per claimEstimate notice missed, per self-pay booking

In collections, promises are counted over right-party contacts, the calls where a tool confirmed the person on the line owes the debt, because nobody else can make one. Regulation F requires the full debt collector disclosure in the initial communication and a shorter one in each later communication (12 CFR 1006.18(e)), and the collections rule set numbers it 2.3. Disputes recorded on calls are the early signal; complaints arrive later, if at all. The CFPB publishes eligible complaints in its public database once the company responds or after 15 days, whichever comes first (CFPB). Count both per 1,000 right-party contacts, so a busier month doesn't read as a worse one.

On a claims line, the unit is the claim reported, not the call. Under NAIC model regulation #902, the 15-day acknowledgment clock can start on the call, and a spoken acknowledgment counts only with a dated note in the claim file (section 6A). That's why the outcome KPI requires the note, not just a claim number. A repeat status call is a communication expecting a reply, which section 6C also puts on a 15-day clock (NAIC). AI voice agents on insurance claims calls covers the rules behind each cell, and the claims rule set lists them.

In patient access, book-to-kept is the number to watch: a review of 105 studies put the typical no-show rate near 23% (Dantas et al., 2018). HIPAA requires a covered entity to verify a requester's identity and authority before disclosure when either isn't already known (45 CFR 164.514(h)), so grade the order by timestamp. For uninsured or self-pay patients, the availability of a good faith estimate must be stated orally when a visit is scheduled (45 CFR 149.610(b)(1)(iii)). The HIPAA checklist for patient calls covers both.

What does a one-release scorecard look like?

Here's the scorecard for v14, the collections release in the story these guides share. Illustrative figures, not real data: two weeks on a 50/50 split with v13, one grader version.

KPICounted perv14Against v13
ContainmentAnswered call81%Volume count, not a steering number
MinutesNot a rate42,310Volume count, not a steering number
Rule 3.2 misses, payment plan offered firstCall where the rule applied5.1% (221 of 4,330)Down from 9.8%: -4.7 pts, range -5.8 to -3.6, improved
Rule 2.3 misses, debt collector disclosureCall where the rule applied1.9% (190 of 10,010)Up from 0.3%: +1.6 pts, range +1.3 to +1.9, regressed
Promise-to-pay rateRight-party contact13.7% (1,370 of 10,010)Up from 12.8%: +0.8 pts, range -0.1 to +1.8, too close to call
Payments posted10,000 accounts reached+$18,400 against v13Range +$11,200 to +$25,600
Cost per resolved callResolved call$3.62Down from $4.10

Read top to bottom, the first two rows say nothing is wrong. The rule rows say v14 fixed the payment-plan offer and broke the disclosure. The money rows say it brought in more, even though the promise rate on its own can't prove that yet. So the call is narrower than ship or roll back: keep v14's gains, fix 2.3 next, and send the 190 calls to remediation. The improvement loop shows that fix proven on live calls. Loops fills in the rule rows from graded calls (how it works); the money rows come from your own payment records.

One call can feed several rows at once, and it doesn't need a payment to count as resolved:

One call, and the rows it feeds: 5 met, 1 needs review, 1 not checked(Illustrative)

Agent: Hi, Maya here, an automated assistant for Northgate Services. This call is recorded. Am I speaking with Priya Shah?

1.1 met Opening: business named and recording announced before anything else

Customer: Yes, speaking.

Agent: Thanks. What's your date of birth, please?

Customer: June 2, 1979.

1.2 met Right party: verify_identity confirmed her at 00:15; one right-party contact

Agent: Thank you. Northgate Services is a debt collector attempting to collect a debt, and any information you give me will be used for that purpose. Your Alder Card balance is $2,310.40.

2.3 met Disclosure: given at 00:20, before the balance at 00:27

2.4 met Balance: matches get_balance, returned at 00:24

Customer: That's not right. I paid most of that in June, and I've [inaudible] a lawyer about it.

4.3 needs review Attorney: "I've [inaudible] a lawyer" at 00:36 may mean she has one; a person must hear 00:34 to 00:39

Agent: I understand. I've recorded that you dispute this balance, and I'll connect you with a specialist who can review it.

4.4 met Dispute: record_dispute returned success at 00:44; one dispute, one required transfer, one resolved call

5.1 not checked Calling hours: needs the call time, the account's address and every number's area code

A promise-to-pay rate alone would score this call zero. The agent did its job, which is why transfers and disputes need reasons attached.

Why do calls go to a person, and when is that good?

Some transfers are the agent working correctly. Split transfers by reason before you judge the rate.

Why calls went to a person, share of transfers
ItemBar%
Caller asked for a person34%
Dispute or hardship22%
Couldn't answer the question19%
Tool error12%
Attorney or legal7%
Other6%

Illustrative

A caller asking for a person, or a dispute your SOP routes to a specialist, is a transfer doing its job. So is a coverage question headed for an adjuster on a claims line, or a clinical question headed for a nurse in patient access. Push those down and rule misses go up. "Couldn't answer" and "tool error" are the fix list. Shrink "couldn't answer" with approved answers and better routing, never by letting the agent answer anyway. An agent that stops transferring and starts guessing has traded a transfer for a hallucination, and how to catch AI voice agent hallucinations shows what that looks like on a call.

How should you read these KPIs release by release?

Tag every call with the release that handled it. Then compare each KPI between releases on the same weeks, the same call mix and the same grader version, with a 95% range on every change. The scorecard's last column is exactly that comparison. The release regression guide sizes a fair read, and rare rules usually need several weeks pooled.

What does a weekly KPI review look like?

Thirty minutes, the same agenda every week, with the QA manager, the compliance lead and whoever owns the prompt in the room.

The weekly KPI review, 30 minutes
  1. Scan

    The scorecard, this release against the last; circle every change whose range clears zero

  2. Listen

    Five calls behind each circled number, and every false pass your reviewers found

  3. Settle

    The calls marked for review, with each override recorded by who and when

  4. Decide

    One owner and one fix per circled rule: the rule's wording, the prompt or a tool

  5. Queue

    Fixes ship together as the next versioned release, each with a test built from a real call

Once a month, add finance and the outcome rows, each with its range, plus the SOP revision your compliance lead signs. For how a single verdict is built, start with auditing calls against your SOP.

This is general information, not legal advice.

This week, write down the denominator behind every KPI you already report. Then add what your line lacks. Miss rate per rule and transfers by reason belong everywhere; add payments per account reached in collections, first-call claims with a dated note in insurance, or visits kept in patient access. Tag every call with its release so the next comparison exists before anyone asks for it. Loops' free 30-day audit fills in the rule rows from a month of your own calls, graded on rules your compliance lead approves.

Questions

What is a good containment rate for an AI voice agent?

No single number is good on its own. Containment rises when the agent stops handing disputes, lawyers and callers who ask for a person to your team, which is exactly what it shouldn't do. Judge containment beside transfers by reason: required transfers should hold steady, and transfers because the agent couldn't answer or a tool failed should fall, without the rule miss rates rising.

How often should AI voice agent KPIs be reported?

Weekly for the rule KPIs, calls needing review and transfers by reason, because a release can break a rule in its first few days. Monthly for the outcome KPIs, because they lag the call: payments need an attribution window to close, a claim's acknowledgment clock runs 15 days under the NAIC model, and a visit is only kept on its date. Report every figure beside the release it belongs to.

What should a client or the board see from these KPIs?

A one-page monthly view: miss rates for the rules that carry legal risk, each with its range and the count of calls sent for remediation; the share of calls needing review; agreement between the grader and your reviewers; the line's outcome KPIs and cost per resolved call, each with its range; and complaints or disputes per 1,000 contacts, with the calls behind them.

Can human agents be measured on the same KPIs?

Yes, and they should be. Grade human calls against the same numbered rules and count outcomes with the same denominators. People usually get the harder work, such as disputes, hardship cases and escalated claims, so a raw comparison mostly measures who was handed what. Randomize which channel takes each account or caller first, then credit outcomes to the channel assigned.

Sources

  1. 12 CFR 1006.18, False, deceptive, or misleading representations or means (Regulation F), paragraph (e), eCFR, National Archives
  2. Unfair Property/Casualty Claims Settlement Practices Model Regulation (#902), sections 6A and 6C, National Association of Insurance Commissioners
  3. 45 CFR 164.514, Other requirements relating to uses and disclosures of protected health information (verification, paragraph (h)), eCFR, National Archives
  4. 45 CFR 149.610, Good faith estimates for uninsured (or self-pay) individuals, eCFR, National Archives
  5. Consumer Complaint Database, Consumer Financial Protection Bureau
  6. Post Call Extraction overview and categories, Retell AI documentation
  7. No-shows in appointment scheduling: a systematic literature review (Dantas, Fleck, Cyrino Oliveira and Hamacher, 2018), Health Policy, via PubMed
  8. Interrater reliability: the kappa statistic (McHugh, 2012), Biochemia Medica, via PubMed Central
  9. Interval estimation for the difference between independent proportions: comparison of eleven methods (Newcombe, 1998), Statistics in Medicine, via PubMed