How to measure what an AI voice agent release earned
Short answer
Measure AI voice agent ROI from outcome records, not minutes or containment. In collections, credit each payment to the release that handled the account's last call before it posted, within a fixed window such as 30 days. Compare releases on the same weeks, clients, and balance and age bands, put a 95 percent range on every difference, and count staff time in cost per resolved call.
On this page
- What should you measure instead of minutes and containment?
- How do you credit a payment to a release?
- How do you compare two releases like with like?
- When is a difference too small to call?
- What does a release comparison look like in numbers?
- Why track promise-to-pay and kept-promise rates separately?
- What does a resolved call cost?
- How do you compare the AI agent with human agents fairly?
- What does "earned" mean on a claims line or a patient-access line?
- Which pitfalls make the numbers wrong?
- Questions
- Sources
Your voice platform reports minutes and containment, the share of calls the agent finished without handing off to a person. Finance asks something else after every release: did this version bring in more than the last one? A contained call can end in a hang-up with no money moving, so the answer comes only from joining calls to payment records on comparable accounts. This guide works one illustrative collections release, then shows what "earned" means on a claims line and a patient-access line.
What should you measure instead of minutes and containment?
| What dashboards report | What finance needs |
|---|---|
| Minutes handled | Dollars posted per 10,000 accounts reached |
| Containment rate | Promises kept, not only promises made |
| Results per call placed | Results per account reached |
| Cost per minute | Cost per resolved call, staff time included |
| A total for this release | The difference from the last release, with a range |
In collections, four measures do the work, and each has exactly one denominator:
- Promise-to-pay rate: promises made per right-party contact. A right-party contact is a call on which a tool confirmed the agent was talking to the consumer who owes the debt.
- Kept-promise rate: promises paid in full within a set grace period, per promise whose date and grace period have passed.
- Dollars posted per 10,000 accounts reached, inside the attribution window. An account counts as reached once the release has placed a call to it in the comparison weeks, whoever answered. Count only amounts the agreement creating the debt or the law authorizes. For a debt collector, any other fee violates 12 CFR 1006.22(b), so it's a refund waiting to happen, not revenue.
- Cost per resolved call, covered below.
Why accounts reached, and not right-party contacts, for the money? Because a release can change how many calls get that far, and the one in the example below shortened its opening to cut early hang-ups. Calls placed is no better: how often you dial is your strategy, limited by the call-frequency presumption in Regulation F, the CFPB's debt collection rule (12 CFR 1006.14(b)(2)).
How do you credit a payment to a release?
Pick one attribution rule, write it down and use it for every comparison. An attribution window is the number of days after a call during which a payment can still be credited to that call.
- Credit last touch
A payment counts toward the release that handled the account's last call before it posted, if that call falls inside the window
- Fix the window
Long enough to span the paydays your consumers get; write it down before the comparison begins
- Try a shorter window
14 days tests whether the result leans on slow payments the call may not have caused
- Try any-touch
Crediting every release that spoke with the account shows whether a rollout put two releases on the same accounts
Thirty days is a sensible first window because of how people are paid. In February 2023, 43.0% of US private establishments paid every two weeks and 27.0% every week (BLS). So for about 70% of employers, 30 days holds at least two paydays, and the semimonthly and monthly payers who make up the rest usually get one. If the ranking of two releases flips between methods, the difference isn't solid enough to report.
Last touch also credits some self-cures (accounts that would have paid anyway) and payments that followed a letter or a text, so every release's total looks better than it is. The difference between two releases on comparable accounts holds up far better, because both sides get the same letters and self-cures.
How do you compare two releases like with like?
The cleanest design runs both releases at the same time on a random split of accounts, for example by a hash of the account ID, so both see the same weeks, clients and paydays. When releases ran one after the other, match instead: compare within the same client, balance band and age band, then weight each band's difference to a common mix.
Age alone moves results a long way. The FTC's study of debt buyers estimated, from its regression model, what they paid for credit card debt by age:
Debt past 15 years sold for virtually nothing, so a release that happened to work fresher placements looks better for reasons unrelated to the release.
Pooling can even reverse the answer (Simpson's paradox): release B can beat release A in every age band and still lose overall if it was handed more old accounts. Compliance comparisons need the same matching, which the guide to release regressions on live calls works through.
When is a difference too small to call?
When its range includes zero, or is so wide that "it cost us money" is still plausible. For a rate, use an interval built for the difference of two proportions. Newcombe's comparison of eleven methods found the simple textbook formulas prone to poor coverage, and recommended combining the Wilson score interval for each side, which works at any sample size (Newcombe, 1998).
Dollars need a different tool, because payment amounts are skewed: a handful of settlements in full can move an average more than a thousand small installments. Resample instead (a bootstrap). Draw each side's accounts with replacement a few thousand times, recompute the difference each time, and report the middle 95 percent of the results.
Fix the length of the comparison before you look; stopping on the first good morning finds differences that aren't there. A range that crosses zero still answers something: the release hasn't earned or cost enough to see yet.
What does a release comparison look like in numbers?
Illustrative example, not real data, from the release story these guides share. Release v14 cut its opening short to reduce early hang-ups and offered the payment plan sooner. It ran two weeks against v13 on a 50/50 split by account, about 200,000 accounts reached a side, with the same clients. Payments were credited by last touch within 30 days and read once every window had closed.
| Rule | Interval | Change and read |
|---|---|---|
| Promise to pay: 12.8 to 13.7% | +0.8 pts (−0.1 to +1.8) Too close to call | |
| Kept promise: 62.9 to 61.8% | −1.2 pts (−4.9 to +2.5) Too close to call |
Illustrative. Promise to pay per right-party contact (1,290 of 10,040, then 1,370 of 10,010); kept promise per promise due (812 of 1,290, then 846 of 1,370). Newcombe 95% intervals.
Neither rate settles it. v14 won 0.8 more promises per 100 right-party contacts, but with about 10,000 contacts a side the range still reaches below zero. Its kept rate moved within noise. The dollars are counted over every account reached, not only the ones that got as far as a promise, so they pick up every path to a payment at once:
+$18,400
more posted per 10,000 accounts reached by v14 than by v13
Illustrative
+$11,200 to +$25,600
the 95% range, from resampling about 200,000 accounts a side
Illustrative
$4.10 to $3.62
cost per resolved call, v13 to v14, staff time included
Illustrative
So v14 earned more, by at least $11,200 per 10,000 accounts on this evidence. The same release skipped the debt collector disclosure on 190 calls (rule 2.3 in the example collections SOP), and a revenue report that leaves that out describes half a release. The improvement loop follows how v15 fixed it.
Why track promise-to-pay and kept-promise rates separately?
Because a release can raise one by lowering the other. An agent tuned to close harder wins promises from people who can't keep them, so promises climb while payments don't. Fix the grace period (three days, say), and measure kept promises only once each date and grace period has passed, or the newer release is judged on an incomplete count.
Read one rule beside the kept rate: whether the promise was logged. A promise the agent failed to record is still one the consumer made, and it's the one least likely to get a reminder or a payment link. That's rule 4.1 in the example SOP, and v14 cut its miss rate from 6.5% to 3.2%.
Pressure has a compliance cost too. Implying consequences the collector can't lawfully take, or doesn't intend to take, is a representation 12 CFR 1006.18 prohibits. A lender collecting its own debts answers to the ban on deceptive or abusive acts in 12 U.S.C. 5536(a)(1)(B). Revenue earned that way comes back as remediation. How to audit AI voice agent calls against your standard operating procedure (SOP) covers the grading, and FDCPA and Regulation F rules for AI collection calls covers the rules. The collections rule set shows each one numbered beside the regulation it comes from.
What does a resolved call cost?
Cost per resolved call divides the total cost of a set of calls by the calls that reached an intended end. An intended end is a payment taken, a promise made, a dispute or cease request recorded, or a transfer to the right person. It beats cost per minute, because a cheap minute that resolves nothing is expensive.
Count everything the release causes, people included. In the example, v14's $3.62 per resolved call splits into $2.00 of platform, telephony and model charges, $1.17 of staff time on calls the agent transferred and $0.45 of review time on flagged calls. Staff minutes are costed at a loaded $0.75 each (Illustrative). A report built from the platform bill alone shows $2.00, which understates the true cost by 45%.
How do you compare the AI agent with human agents fairly?
Your human floor handles callbacks, disputes, hardship cases and the largest balances, so a raw comparison mostly measures the work each side was handed. For a fair read:
- Assign accounts at random to AI-first or human-first for a set period, within the same client, balance and age bands.
- Credit outcomes to the assigned channel, AI calls that transferred included. This is the intention-to-treat principle from clinical trials: judge the strategy you chose, not only the calls that went smoothly.
- Use the same attribution window, measures and cost accounting on both sides.
- Grade both sides on the same SOP rules, so a revenue gain bought with compliance misses shows up next to the gain.
The gap between the two groups is the agent's return on investment (ROI): dollars posted per 10,000 accounts, AI-first minus human-first, set against the difference in total cost, each with its range. It's the one ROI figure that survives the question "compared with what?"
What does "earned" mean on a claims line or a patient-access line?
The method carries over where the agent collects no money. Pick the outcome the line exists for, count it per unit the release can't change, and put staff time in the cost.
An insurance claims line
On first notice of loss (FNOL) and claim-status calls, the return is staff time and deadlines met, and the unit is the claim reported, not the call. Model regulation #902, the NAIC model many states' claims rules follow, gives an insurer 15 days to acknowledge a claim. An acknowledgment that isn't in writing needs a dated note in the claim file, and notice to the insurer's agent counts as notice to the insurer (section 6A). Other claimant communications that reasonably expect an answer get a reply within 15 days (section 6C) (NAIC).
So count the claims that get a claim number and a dated acknowledgment note on the first call, and count repeat calls per claim, since each is a question somebody has to answer. Illustrative: say a release cuts repeat calls from 38 to 31 per 100 claims, and each takes a person 8 minutes at a loaded $0.90 a minute. That saves about $500 of staff time per 1,000 claims. Subtract what its misses cost. A coverage answer given without a policy lookup is exposure, not savings; AI voice agents on insurance claims calls explains why, and the claims rule set lists the rules.
A healthcare patient-access line
On a scheduling line, count visits kept per 1,000 patients reached, not visits booked. Across 105 studies, the average no-show rate was about 23%, and the most commonly reported predictors were a long wait for the appointment and a history of no-shows (Dantas et al., 2018). A release that fills the calendar further out can book more visits and keep fewer. On a billing line, dollars posted per 10,000 accounts reached works as it does in collections.
Two rules sit beside either number. The caller's identity and authority are verified before any protected health information, a balance included (45 CFR 164.514(h)). And an uninsured or self-pay patient hears, when a visit is scheduled or cost comes up, that a good faith estimate is available (45 CFR 149.610(b)(1)(iii)). HIPAA and AI voice agents in patient access covers both.
Which pitfalls make the numbers wrong?
Start with seasonality. By March 13, 2026, the IRS had issued 50.4 million refunds worth $182.6 billion (IRS), and by May 8, 99.1 million worth $324.8 billion (IRS). A release shipped in February and one shipped in June faced different money.
- Both releases ran in the same weeks, or a group that didn't change over those weeks checks for season
- Same clients, balance mix and placement batches, compared within bands
- The account set frozen for the test, or recalled accounts dropped from both sides
- Self-cures measured, where your strategy allows, by a small holdout that gets no AI call
- No new dialing schedule, settlement offer or letter in the same week, or its effect separated from the release's
This is general information, not legal advice.
Ahead of your next release, write down the attribution rule and window, and tag every call with the agent version. Then set up a daily payments file (account ID, amount, date posted), so the answer exists when someone asks what the release earned. Loops doesn't read payment files, but it supplies the other half: each rule's miss rate, release against release, on live calls (how it works). You can start with a free 30-day audit.
Questions
What attribution window should a collections team use?
Start from your consumers' paydays. Most US private employers pay weekly or every two weeks, so 30 days after the last call usually spans at least two paydays and catches a payment made on a promised date later in the month. Fix the window before the comparison starts and rerun it at 14 days. If two releases swap places when the window changes, the difference between them isn't solid enough to report.
How many accounts do you need to compare two AI agent releases?
It depends on the base rate and the size of difference you care about. In the illustrative v13 to v14 comparison, about 10,000 right-party contacts a side left a range of roughly plus or minus 0.9 points on the promise-to-pay difference, too wide to confirm a 0.8-point gain. The dollar comparison ran over about 200,000 accounts a side and cleared zero. Plan the sample from the smallest change worth acting on, before the test starts.
Should payment plans count at full value or as installments arrive?
Count dollars as they post inside the attribution window, and report scheduled plan value separately. A plan set up on a call is a promise until the money arrives, and counting future installments as earned rewards a release that sets up plans consumers later abandon. The kept rate on first installments is the early signal of plan quality.
Can you measure AI voice agent ROI without a random split between releases?
Yes, with more care and less certainty. Compare within the same client, balance band and age band, weight each band's difference to a common mix, and use a group that didn't change over the same weeks, such as your human floor, as a check on seasonality. Report the result as weaker evidence than a randomized split would give.
What does ROI mean for an AI agent on a claims or scheduling line?
The same comparison with a different outcome. On a claims line, count claims opened on the first call with a dated note and repeat calls per claim, and put a dollar value on the staff minutes saved. On a scheduling line, count visits kept per 1,000 patients reached rather than visits booked. In both, set the gain against total cost with people's time included, and read it beside the rules that could cancel it.
Sources
- 12 CFR 1006.18, False, deceptive, or misleading representations or means (Regulation F), eCFR, National Archives
- 12 CFR 1006.22, Unfair or unconscionable means (Regulation F), eCFR, National Archives
- Unfair Property/Casualty Claims Settlement Practices Model Regulation (#902), sections 6A and 6C, National Association of Insurance Commissioners
- 45 CFR 164.514, Other requirements relating to uses and disclosures of protected health information (verification, paragraph (h)), eCFR, National Archives
- 45 CFR 149.610, Good faith estimates for uninsured (or self-pay) individuals, eCFR, National Archives
- The Structure and Practices of the Debt Buying Industry (January 2013), Federal Trade Commission
- Length of pay periods in the Current Employment Statistics survey (February 2023), U.S. Bureau of Labor Statistics
- Interval estimation for the difference between independent proportions: comparison of eleven methods (Newcombe, 1998), Statistics in Medicine, via PubMed
- No-shows in appointment scheduling: a systematic literature review (Dantas, Fleck, Cyrino Oliveira and Hamacher, 2018), Health Policy, via PubMed
- Filing season statistics for week ending March 13, 2026, Internal Revenue Service