Why a voice agent release that passes its tests can still break a rule
Short answer
Voice agent regression testing checks that a new release doesn't break rules the last one followed. Simulated suites catch many breaks but can't reproduce real callers, live tool failures or every rule a prompt change touches. So tag every live call with its agent version and compare each rule's miss rate between versions on the same weeks, call mix and grader version, with a confidence interval.
On this page
- Why do voice agent tests pass while live calls fail?
- What do Retell, Vapi, Bland, ElevenLabs, LiveKit and Pipecat provide?
- How do you tag every call with the agent version?
- How do you compare two releases fairly on live calls?
- What does a fair comparison look like in practice?
- What should you do when a release regresses?
- Questions
- Sources
Picture the release your team shipped last Tuesday. It passed every scenario in its test suite, and early hang-ups are down. Ten days later a QA reviewer hears the agent read a consumer's balance before saying it's a debt collector, and nobody can say how many other calls did the same. The suite wasn't wrong; it never met the callers who broke the rule. And nothing compared the new release with the old one on live calls.
Why do voice agent tests pass while live calls fail?
A suite only meets the callers, phrasings and tool responses its authors wrote. A regression, in this sense, is a rule the previous release followed that the new release breaks more often.
| A simulated suite tests | Live calls show |
|---|---|
| Callers the agent's builders wrote | Consumers who ask "who is this?" first |
| Text runs that skip speech recognition | What speech recognition actually heard |
| A few dozen scenarios | The long tail of real calls |
| One pass per scenario | How much answers vary run to run |
| Mocked tools that always succeed | Tool calls that time out in production |
| Whether the disclosure appears | Whether it came before the balance |
- Callers: "synthetic callers do not fully represent real callers and their environments," as Vapi's testing guide puts it. Retell's simulation tests run as text, and LiveKit and Pipecat default to text mode, which skips speech recognition.
- Coverage: if 50 scenarios all pass, the most you can claim at 95% confidence is a failure rate below about 6% on those scenarios, by the rule of three (Hanley and Lippman-Hand, JAMA 1983). On an agent taking 20,000 calls a week, a rule missed on 0.5% of them is 100 consumers.
- Consistency: τ-bench is a 2024 benchmark of tool-using agents following domain rules. On it, gpt-4o succeeded on fewer than half the tasks, and in retail it got all eight tries of a task right (pass^8) under 25% of the time (Yao et al.). Vapi recommends several iterations for critical scenarios "because one pass does not prove consistent behavior."
- Tools: ElevenLabs and Vapi let tests mock tool responses, so the payment API never times out; in production it does, and the agent may say a promise was recorded when it wasn't. Vapi warns that "a tool reporting success does not independently prove that an external system changed."
- Side effects: formatting changes that kept a prompt's meaning identical moved few-shot accuracy by up to 76 points on LLaMA-2-13B (Sclar et al., ICLR 2024). And under the same product name, GPT-4's accuracy at identifying prime numbers fell from 84% in March 2023 to 51% in June 2023 (Chen, Zaharia and Zou).
What do Retell, Vapi, Bland, ElevenLabs, LiveKit and Pipecat provide?
All six let you test before release, and most hosted ones put a version on each call record. This is what their public docs describe as of September 2026.
| Platform | Pre-release testing in the docs | Versions on live calls |
|---|---|---|
| Retell | Simulation tests (an AI caller in a text conversation, graded pass or fail), batch regression runs, web and phone test calls | Read-only published versions; agent_version on each call |
| Vapi | Evals (mock conversations at the text and model layer); Simulations in chat or voice | Published versions; outbound and web calls can pin assistantVersion |
| Bland | Text and voice test chat, per-node unit tests, Standards from the testbed | Staging and production versions; pathway_version on each call |
| ElevenLabs | Simulation, next-reply and tool-call tests, with tool mocking | Immutable versions, branches and traffic splits; version_id on each conversation |
| LiveKit | Unit tests in pytest or Vitest; Agent Simulations in text or audio (beta) | lk agent deploy rolls out versions; tag sessions yourself |
| Pipecat | Pipecat Evals, scripted or simulated, LLM-judged, in text or audio | Open-source; tag calls yourself |
Run these on every change. Three also grade live calls against criteria you write: Retell's AI QA on a sample you size, Vapi's Scorecards with points per metric, and ElevenLabs' success evaluation with up to 30 criteria per agent. ElevenLabs' Experiments go furthest, splitting live traffic between branches and warning that "small sample sizes lead to unreliable results."
None of these pages describe the live half of voice agent regression testing. That's a per-rule comparison of two releases, with an interval on each change and the failing calls listed, graded the same way on any platform and on human calls.
How do you tag every call with the agent version?
Use the platform's version field where one exists (agent_version on Retell, pathway_version on Bland, version_id on ElevenLabs), and stamp your own release ID where it doesn't. The tag must cover everything that changes behavior: prompt and flow, model and snapshot, tool definitions, knowledge base, voice and transcriber settings. Tools are the usual gap: a Vapi tool set to Latest follows its newest published version, so a tool change can reach callers without a new assistant version.
- On LiveKit, Pipecat or an in-house stack, stamp a release ID at call start: a git commit plus a hash of the prompt, model name and tool schemas.
- Log the model name your provider returns on each response, so a silent model update shows up.
- Never infer the version from the timestamp. LiveKit's rolling deploys give old instances up to an hour to finish active sessions (LiveKit), and traffic splits run two versions at once by design.
How do you compare two releases fairly on live calls?
Compare miss rates rule by rule. Each rule is a numbered line from your standard operating procedure (SOP) that compliance has approved; how to audit calls against your SOP shows how to write them. A rule's miss rate is the share of calls where the rule applied and the agent missed it. A disclosure rule applies to calls that reached the right party, and a promise-logging rule only when the consumer promised.
- Same weeks
Run both releases on the same days, ideally as a split by a hash of the account ID.
- Same mix
Compare within segments, then weight both sides to a common mix.
- Same grader
Grade both periods with one grader version and the same rule wording.
- Enough calls
Set each rule's call count before launch, and run to it.
When accounts take several calls, resample accounts rather than calls (a cluster bootstrap) for the interval. On outbound calls you can pick the version: Vapi takes assistantVersion, Bland a pathway version number, ElevenLabs a branch. A before-and-after comparison needs adjacent weeks, away from month-end, holidays and new portfolio placements.
Segment by direction, client, account age and first contact or repeat, because the mix alone can move the total with no change in the agent:
A grader upgrade or a reworded rule between periods mixes the grader's change into the agent's. So regrade both periods with one version, and keep ElevenLabs' per-branch evaluation criteria identical. More in grading calls with your own LLM judge.
How many calls, and when is a change real?
The rule for a release decision is short. Decide before launch how many calls each rule needs, and run to that number instead of stopping the moment a gap looks big. Then call a rule regressed only when the whole range around its change sits above zero. If the range includes zero, the honest verdict is "too close to call," not "no change."
Small changes in rare misses need the most calls: catching a rise from 0.5% to 1% takes about 4,700 calls a side, while a rise from 3% to 6% shows up in about 750. The range is a 95% confidence interval, the set of changes your data can't rule out. And with 40 rules each checked this way, about two will look like they moved even if nothing changed, so confirm a surprise on next week's calls before acting on it.
For the analyst: the counts above give 80% power, an 80% chance of detecting a rise that size when it's real, at a 5% two-sided significance level. Use a Wilson score interval for each rate rather than the textbook formula, whose lower limit can go below zero (NIST's handbook covers both). Combine the two into Newcombe's hybrid score interval for the difference (Newcombe, 1998). Switch to Fisher's exact test when a rule applied on only a few dozen calls (NIST).
This is the comparison Loops runs on live calls: each rule's miss rate between releases, the same grader version on both sides, and the calls behind any change listed (how the grading and release comparison work).
What does a fair comparison look like in practice?
This example uses illustrative numbers and the fictional collections SOP every guide on this blog shares, so rule 2.3 is always the debt collector disclosure. Release v14 shortened the opening to cut early hang-ups and moved the payment-plan offer earlier. It passed its suite, then ran two weeks on a 50/50 split with v13 by account, same clients, one grader version.
| Rule | Interval | Change and read |
|---|---|---|
| 1.2 Right party | −0.6 pts (−0.9 to −0.3) Improved | |
| 2.3 Disclosure | +1.6 pts (+1.3 to +1.9) Regressed | |
| 3.2 Payment plan | −4.7 pts (−5.8 to −3.6) Improved | |
| 4.1 Promise logged | −3.3 pts (−5.0 to −1.7) Improved | |
| 4.2 Stop calling | +3.2 pts (−0.6 to +7.3) Too close to call |
Illustrative. Newcombe hybrid score 95% intervals; two weeks, 50/50 split by account, one grader version.
Illustrative counts, as misses out of the calls where each rule applied:
| Rule | v13 misses | v14 misses |
|---|---|---|
| 1.2 Right party | 1.4% (163 of 11,640) | 0.8% (94 of 11,720) |
| 2.3 Disclosure | 0.3% (30 of 10,040) | 1.9% (190 of 10,010) |
| 3.2 Payment plan | 9.8% (417 of 4,260) | 5.1% (221 of 4,330) |
| 4.1 Promise logged | 6.5% (84 of 1,290) | 3.2% (44 of 1,370) |
| 4.2 Stop calling | 2.9% (7 of 245) | 6.0% (14 of 232) |
Newcombe's interval combines the two Wilson intervals and treats calls as independent. If accounts took several calls, a cluster bootstrap would widen every interval, but the 2.3 interval would have to grow nearly six times wider to reach zero.
v14 fixed what it set out to fix and broke rule 2.3. Rule 4.2 doubled on paper, but with about 240 calls a side its interval spans zero. Detecting a rise from 3% to 6% takes roughly 750 calls a side. Here's one of the 190 calls behind the regression.
Agent: Hi, this is Maya, an automated assistant calling for Quillfeather Services on a recorded line. Am I speaking with Luis Ortega?
1.1 met Opening: business named and recording announced in the first turn
Customer: Who is this? What's this about?
Agent: I can explain once I confirm I'm speaking with Luis. Can you confirm your date of birth?
Customer: July 19, 1990. So what is it?
1.2 met Right party: verify_identity returned a match at 00:19
Agent: Thanks, Luis. You have a balance of $2,316.40 with Alder Card, and I can set up a payment plan today.
2.4 met Balance: $2,316.40 matches what get_balance returned at 00:22
2.3 missed Disclosure: balance stated at 00:24; disclosure not given until 00:41
3.2 met Payment plan: plan offered before any request for the full balance
Customer: I could do something small on the 3rd.
Agent: Quillfeather Services is a debt collector. This is an attempt to collect a debt, and any information obtained will be used for that purpose. I can set you up for $150 on October 3.
Customer: Okay, the 3rd works.
Agent: You're set for $150 on October 3.
4.1 missed Promise logged: log_promise called at 00:52 and returned a timeout; no retry
5.1 not checked Calling hours: needs the call time, the account's address and every number's area code
The suite's disclosure test passed because its caller never asked why the agent was calling, and its criterion checked that the disclosure appeared, not that it came before the debt.
What should you do when a release regresses?
List the calls, fix with compliance, and prove the next release live. First, make sure the regression is real.
- Every call carries a release ID, not a date guess
- Both releases ran the same weeks, or adjacent ones
- Compared within segments, weighted to one mix
- One grader version and rule wording on both sides
- The rule reached the call count set before launch
- The 95% interval on the change excludes zero
- A surprise held up on the next week's calls
- The calls behind the change are listed
The list is the work, not the percentage. Every call that missed rule 2.3, with its quoted line and timestamp, goes to whoever owns remediation.
Regulation F is the Consumer Financial Protection Bureau's rule under the Fair Debt Collection Practices Act. It requires the full disclosure in a collector's initial communication with a consumer, and a statement that the communication is from a debt collector in each later one (12 CFR 1006.18(e)). Collectors must also keep records that evidence compliance or noncompliance until three years after the last collection activity on a debt, and call recordings for three years (12 CFR 1006.100). Keep the calls, verdicts and release ID together; more in FDCPA and Regulation F rules for AI collection calls.
All five hosted platforms above can put a previous version back (LiveKit Cloud's instant rollback needs a paid plan). Fix the order in the prompt, and make the test criterion check order, not presence. Turn the failing calls into tests: Retell, ElevenLabs, LiveKit and Bland each document a path from a real call to a test. Then prove v15 against v14 on rule 2.3, confirm the gains on 1.2, 3.2 and 4.1 held, and finish collecting on 4.2. Whether the regression cost money is covered in how to measure what a voice agent release earned.
This is general information, not legal advice.
Before your next release ships, stamp the version on every call, pick the five rules you'd least like to break, and write down how many calls and what interval will count as a regression. If you'd like to see that comparison on your own calls first, Loops offers a free 30-day audit.
Questions
Should we keep running simulated tests if we compare releases on live calls?
Yes. Simulated suites catch many breaks before any consumer hears them, and they're cheap to run on every change. The live comparison catches what they can't reproduce: real callers, tool failures in production and rules a prompt change quietly moved. When a live call exposes a gap, turn it into a new test case so the suite catches that pattern next time.
How long should a new release run before we compare it with the last one?
Until each rule you care about reaches the call count you set before launch. Detecting a rise from 0.5% to 1.0% takes roughly 4,700 calls per version where the rule applied, at 80% power and 5% significance; a rise from 3% to 6% takes about 750. Rare rules may need several weeks pooled. Run to that count rather than stopping when a gap looks big.
Can we compare two releases if we upgraded our grader in between?
Not directly, because a change in miss rate would mix the agent's change with the grader's. Regrade the earlier release's calls with the new grader version, or keep the old grader for both, so each side is scored the same way. The same goes for a rule whose wording changed: compare it only across periods graded with the same text.
Is a canary or traffic split better than a before-and-after comparison?
Usually, yes. A concurrent split gives both versions the same days, clients and call mix by design, which removes most of what a before-and-after comparison has to adjust for. Split by a hash of the account ID so each consumer hears one release. The trade-off is volume: a 5% canary collects calls slowly, so intervals on rare rules stay wide until you widen the split or wait longer.
Doesn't our voice platform's own QA already score live calls?
Often it does, and it's worth using. Retell's AI QA, Vapi's Scorecards and ElevenLabs' success evaluation all grade calls after they end, against criteria you write. A release decision needs something narrower: each rule's miss rate on both releases over the same weeks, with one grader version, an interval on the change and the failing calls listed. Check whether your platform's reports give you that before you ship on them.
Sources
- 12 CFR 1006.18, False, deceptive, or misleading representations or means (Regulation F), eCFR
- 12 CFR 1006.100, Record retention (Regulation F), eCFR
- e-Handbook of Statistical Methods, 7.2.4.1 Confidence intervals, NIST/SEMATECH
- e-Handbook of Statistical Methods, 7.3.3 Comparing two proportions, NIST/SEMATECH
- Interval estimation for the difference between independent proportions: comparison of eleven methods (Newcombe, 1998), Statistics in Medicine, via PubMed
- Testing voice agents, Vapi
- Scorecard quickstart, Vapi
- Automatically evaluate call quality with AI QA, Retell AI
- Success evaluation, ElevenLabs
- Experiments, ElevenLabs
- τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (Yao et al., 2024), arXiv