The improvement loop: how to improve an AI voice agent after launch

Launch week gets the attention. After that, the prompt gets edited after a complaint, the model under it changes, callers find paths nobody scripted, and the dashboard keeps counting minutes and containment. Six months in, nobody can say whether release 14 is better than release 13, which rules it follows, or what it earns. Gartner expects over 40% of agentic AI projects to be canceled by the end of 2027, citing escalating costs, unclear business value or inadequate risk controls (Gartner, June 2025).

An improvement loop is a fixed cycle: every change to the agent is checked on live calls against the same approved rules from your standard operating procedure (SOP), and every result is written back to one shared record. It has six stations and one hub.

1. Writethe SOP as numbered rules
2. Approvecompliance signs each rule
3. Shipa versioned release
4. Checkevery call, with evidence
5. Provethis release vs the last
6. Improvefix the rule or the prompt

Then back to 1. Every pass writes back to Approved rules and evidence.

The ring is the work. The hub is what the work leaves behind: rule versions and their approvers, verdicts with quoted lines, release comparisons, reviewer overrides and the fixes that followed. Each pass starts from everything the last ones learned, which is why the loop gets cheaper to run.

Why do AI voice agents stop improving after launch?

Because only the easy half of the loop runs. Teams ship and they edit, and the checking in between is a handful of calls somebody had time to hear.

Without a loop	With an improvement loop
The prompt is edited after a complaint	The rule is fixed first, then the prompt, then proven
A few calls a week heard by hand	Every call graded on every rule it triggered
"v14 feels better"	Each rule's miss rate, v14 against v13, with an interval
Findings live in chat threads	Findings written back to one shared record
Minutes and containment	Dollars posted per release, with a range

Regulators and standards bodies already describe something like this loop. The CFPB's examination procedures define a compliance management system as how an institution sets its compliance responsibilities, reviews operations to make sure they're carried out, and "takes corrective action and updates tools, systems, and materials as necessary" (CFPB, August 2017). NIST's voluntary AI Risk Management Framework says the same on the engineering side: deployed systems monitored in production (MEASURE 2.4), and "measurable activities for continual improvements" built into system updates (MANAGE 4.2) (NIST AI 100-1).

Who owns each step, and how often does it run?

Each station needs one named owner: the person who says the station is done, even when others do part of the work.

Keep the weekly review to 30 minutes with the QA manager, the compliance lead and the build lead in the room. The monthly adds finance and closes with a signed SOP revision.

What does the Check station look like on one call?

Checking means every call, every rule the call triggered, a verdict that quotes its evidence, and an honest "not checked" for any rule whose records you didn't send. Rule 2.3 below rests on Regulation F, the CFPB's debt collection rule, which requires the full disclosure in a collector's initial communication and a statement that each later one is from a debt collector (12 CFR 1006.18(e)). The example SOP adds the timing: before the debt comes up.

Agent: Hi, this is Maya, an automated assistant calling for Quillfeather Services on a recorded line. Am I speaking with Daniel Reyes?

1.1 met · Opening: business named and recording announced

Customer: Yes, that's me.

Agent: Thanks, Daniel. To confirm it's you, what's your date of birth?

Customer: March 4, 1986.

1.2 met · Right party: verify_identity returned a match at 00:14

Agent: Thank you. Your balance with Alder Card is $1,284.16, 94 days past due, and I can set up a payment plan today.

2.4 met · Balance: $1,284.16 matches what get_balance returned at 00:18

2.3 missed · Disclosure: balance stated at 00:21 with no debt collector disclosure before it

3.2 met · Payment plan: offered before any request for the full balance

Customer: I can do $107 on the 3rd.

Agent: You're set for $107 on October 3.

4.1 met · Promise logged: log_promise returned success at 00:39

5.2 not checked · Call frequency: needs the dial log for Daniel and this debt

A hallucination, such as a balance get_balance never returned, shows up the same way: a miss on 2.4 with the quoted amount beside the system's. How to catch AI voice agent hallucinations covers the rest. The grading method itself is in how to audit AI voice agent calls against your SOP.

How do you prove a release against the last one?

Compare each rule's miss rate between the two releases on live calls from the same weeks and call mix, graded by the same grader version, with a 95% interval on each change. An interval that crosses zero means "too close to call," not "no change." In the release story these guides share, v14 shortened the opening and moved the payment-plan offer earlier, and it passed its tests.

Rule	Interval	Change and read
1.2 Right party		−0.6 pts (−0.9 to −0.3) Improved
2.3 Disclosure		+1.6 pts (+1.3 to +1.9) Regressed
3.2 Payment plan		−4.7 pts (−5.8 to −3.6) Improved
4.1 Promise logged		−3.3 pts (−5.0 to −1.7) Improved
4.2 Stop calling		+3.2 pts (−0.6 to +7.3) Too close to call
	Fewer misses0More misses

Illustrative. Same weeks, same grader version; Newcombe 95% intervals.

v14 fixed three rules and broke one. Rule 2.3 went from 30 misses in 10,040 calls to 190 in 10,010. Those 190 calls are the work: they go to compliance for remediation and to the build team as test cases. Rule 4.2 doubled on paper, but at about 240 calls a side its interval spans zero, so it keeps collecting. Why a release that passes its tests can still break a rule covers sample sizes; what a release earned covers the finance half. Loops runs this comparison on every live call, with the same grader version on both sides and the calls behind each change listed (how it works).

Should you fix the rule or the prompt, and when is a pass done?

Ask about the rule first. A miss has three possible causes, and only one of them is the agent.

• The rule is ambiguous. Reviewers keep overturning the grader, or calls on that rule keep landing in review. Rewrite the rule and have it approved as a new version.
• The rule is wrong for the business. The SOP changes, compliance approves the revision, and calls are graded on the new version from then on.
• The agent broke a clear rule. Change the prompt, flow or tool, turn a real failing call into a test, and ship the change as the next release.

Rule 2.3 on v14 was the third case. The fix put the disclosure back before any account detail, added a test where the caller asks "who is this?" first, and shipped as v15, to be proven against v14 on 2.3 while the other gains hold. Improve still returns to Write, not straight to Ship: even a prompt-only fix starts with someone confirming the rule stands. The CFPB's procedures say appropriate corrective action includes "both correction of programmatic weaknesses and full redress for injured parties." The prompt fix is the first half. The 190-call list is where compliance starts on the second.

A pass is done when the fix is proven on live calls, not when it ships. Print this for the weekly review.

• Rule changes approved and versioned before the release ships Your SOP
• Every call tagged with the release that handled it Practice
• Every call graded on every rule it triggered Practice
• Rules missing their records marked not checked, never passed Practice
• Calls marked for review heard, and overrides recorded with who and when Practice
• Each rule compared with the last release on the same grader version Practice
• Calls behind every regression listed and sent for remediation Practice
• Next fix recorded as a rule change or a prompt change, with its test Practice

For a debt collector, those verdicts are the kind of records that evidence compliance or noncompliance, which Regulation F requires be kept from the start of collection activity on a debt until three years after the last (12 CFR 1006.100(a)). The hub is that record.

How do you start a loop in 30 days with an agent that's already live?

You don't need to pause the agent or rebuild anything. Start from last month's calls.

1. 1Stamp

   Days 1 to 3: tag every call with its release ID and export last month's calls with their tool-call logs

2. 2Write and approve

   Days 4 to 7: split the riskiest SOP sections into numbered rules, and have compliance sign each one

3. 3Baseline

   Days 8 to 14: grade last month's calls on every rule and hear every call marked for review

4. 4Calibrate

   Days 15 to 21: have two reviewers grade a few hundred calls and compare them with the grader, rule by rule

5. 5Prove

   Days 22 to 30: ship one fix as a new release and compare it with the baseline on the same grader version

Start with the rules that carry the most risk or the most money. The example collections SOP these guides share has 14. Add the rest once the first pass runs cleanly.

Where do improvement loops break?

• No version on calls. Without the release on each call, nothing can be compared, and dates won't substitute: rollouts and traffic splits run two versions at once. Use your platform's version field or stamp your own release ID.
• Grading only a sample. Example: an agent takes 10,000 calls a week and misses a rule on 1% of them. A 2% sample holds about two of those misses; if a release doubled the rate you'd expect four, which one week can't tell apart from noise.
• Changing the grader mid-comparison. A new judge model or a reworded rule between v13's weeks and v14's weeks mixes the grader's change into the agent's. Regrade both sides on one grader version, and approve rule changes at release boundaries. Grading calls with your own LLM judge covers grader versioning.
• Fixing the prompt without fixing the rule. The agent learns one reading while the grader and your reviewers keep another, so the number doesn't move, or moves for the wrong reason.

How can you tell the loop is paying off?

Look for these, release over release:

• Targeted rules improve with intervals clear of zero, and stay improved a release later.
• Regressions surface in a release's first week, not through a complaint or a client audit.
• Review rates fall, because the rules got clearer.
• Grader agreement with your reviewers, measured per rule with Cohen's kappa (McHugh, 2012), holds or rises at each calibration.
• The time from a found miss to a proven fix shrinks.
• Dollars posted per 1,000 accounts reached move the right way, or the range says it's too early.

If the same rule regresses twice, the loop is patching prompts and skipping the rule. Which KPIs to track for an AI voice agent covers the operating numbers that sit beside these.

This is general information, not legal advice.

This week, stamp the release on every call and pick the three rules you'd least like to break. Next week, grade last month's calls on them and read every miss. To run the first pass on your own calls, Loops' free 30-day audit grades last month's calls against your approved rules, with the first audit within 72 hours of rule approval.