Why a voice agent release that passes its tests can still break a rule

Picture a release that passes every scenario in its suite and ships on a Tuesday. Ten days later a reviewer notices the agent reading balances before it says it's a debt collector. The suite wasn't wrong. It never met the callers who broke the rule.

Why do voice agent tests pass while live calls fail?

Tests pass while live calls fail because a suite only meets the callers, phrasings and tool responses its authors wrote, and live traffic brings five kinds of trouble a suite rarely reproduces. A regression, in this sense, is a rule the previous release followed that the new release breaks more often.

The callers are different. Scenarios are written by the people who built the agent, and a simulated caller sticks to its persona. Real consumers ask "who is this?" before confirming anything. Vapi's testing guide says it directly: "synthetic callers do not fully represent real callers and their environments" (Vapi). Many suites also run as text: Retell's simulation tests do, and LiveKit and Pipecat default to text mode, which skips speech recognition.

The tail is long and a suite is small. If 50 scenarios all pass, the most you can claim at 95% confidence is that the failure rate on those scenarios is below about 6%, by the rule of three from Hanley and Lippman-Hand's 1983 JAMA paper. On an agent taking 20,000 calls a week, a rule missed on 0.5% of them is 100 consumers.

The same agent answers differently on different runs. In τ-bench, a 2024 benchmark of tool-using agents following domain rules, gpt-4o succeeded on fewer than half the tasks, and its pass^8 in the retail domain (success on all eight tries of a task) was under 25% (Yao et al.). Pipecat's docs agree ("the same agent can answer differently run to run"), and Vapi recommends several iterations for critical scenarios "because one pass does not prove consistent behavior."

Tools fail live and succeed in tests. Test runs often mock tool responses (ElevenLabs and Vapi both support it), so the payment API never times out. In production it does, and the agent may say a promise was recorded when it wasn't. Vapi's guide warns that "a tool reporting success does not independently prove that an external system changed."

A fix to one rule moves another. In an ICLR 2024 study, formatting changes that kept a prompt's meaning identical moved few-shot accuracy by up to 76 points on LLaMA-2-13B (Sclar et al.). The model under your prompt can change too: GPT-4's accuracy at identifying prime numbers fell from 84% in March 2023 to 51% in June 2023 under the same product name (Chen, Zaharia and Zou). A test that only checks whether the disclosure appears will pass a release that moved it after the balance.

What do Retell, Vapi, Bland, ElevenLabs, LiveKit and Pipecat provide?

All six let you test before release, most of the hosted ones put a version on each call record, and Retell, Vapi and ElevenLabs also score live calls against criteria you write. This is what their public docs describe as of September 2026.

Run these on every change. Three platforms also grade calls after they happen. Retell's AI QA evaluates a sampled set of calls, sized by a percentage and a weekly cap you choose, against conditions and metric thresholds you configure, with transcript evidence behind each finding. Vapi's Scorecards grade each completed call from its structured outputs, awarding points per metric. ElevenLabs runs success evaluation on each conversation: up to 30 criteria per agent, each returning success, failure or unknown with a rationale.

ElevenLabs goes furthest toward a release comparison. Its Experiments split live traffic between branches, compare them in a branch-filtered analytics view and warn that "small sample sizes lead to unreliable results." What these pages don't describe is the live half of voice agent regression testing: a per-rule comparison of two releases with an interval on each change and the failing calls listed, graded the same way on any platform and on human calls.

How do you tag every call with the agent version?

Use the platform's version field where one exists, stamp your own release ID where it doesn't, and make the tag cover everything that changes behavior: prompt and flow, model and snapshot, tool definitions, knowledge base, voice and transcriber settings. Tools are the usual gap. In Vapi, a tool set to Latest keeps following its newest published version, so a tool change can reach callers without a new assistant version.

• Platform fields: agent_version on Retell, pathway_version on Bland, version_id on ElevenLabs.
• On LiveKit, Pipecat or an in-house stack, write a release ID into the call record when the call starts: a git commit plus a hash of the prompt, model name and tool schemas.
• Log the model name your provider returns on each response, so a silent model update shows up.
• Never infer the version from the timestamp. LiveKit's rolling deploys give old instances up to an hour to finish active sessions (LiveKit), and traffic splits run two versions at once by design.
• Store the grader version with every verdict too.

How do you compare two releases fairly on live calls?

Compare miss rates rule by rule, where each rule is a numbered line from your standard operating procedure (SOP). A rule's miss rate is the share of calls where the rule applied and the agent missed it. The denominator matters: a disclosure rule applies to calls that reached the right party, and a rule about logging a promise to pay applies only when the consumer promised. Then hold four things steady.

1. Same weeks. A concurrent traffic split is cleanest, because both versions see the same days and clients. Split by a hash of the account ID so each consumer hears one release, and when accounts take several calls, resample accounts rather than calls (a cluster bootstrap) for the interval. On outbound calls you can choose the version yourself: Vapi takes assistantVersion, Bland a pathway version number, and ElevenLabs a branch. Before-and-after comparisons need adjacent weeks, away from month-end, holidays and new portfolio placements.
2. Same mix. Stratify by direction, client, account age and first contact or repeat. Illustrative numbers show why: if the disclosure is missed on 0.5% of outbound and 0.1% of inbound calls under both releases, and v13's weeks were 70% outbound while v14's were 40%, the overall miss rate falls from 0.38% to 0.26% with no change in the agent. Compare within segments, then weight to a common mix.
3. Same grader version. If you upgraded the judge model or edited a rule's wording between periods, the difference mixes the agent's change with the grader's. Regrade both periods with one grader version. ElevenLabs lets each branch carry its own evaluation criteria, so keep them identical when you compare. Our post on grading calls with your own LLM judge covers grader versioning.
4. Enough calls. Detecting a rise from 0.5% to 1.0% with 80% power at a 5% significance level takes roughly 4,700 calls per version where the rule applies; 2% to 3% takes about 3,800, and 3% to 6% about 750. Set these numbers before launch, and don't stop the moment a gap looks big.

A 95% confidence interval for the change is the range of differences your data can't rule out at that level. If it excludes zero, the change is unlikely to be noise. If it includes zero, the honest verdict is "too close to call," not "no change." Use a Wilson score interval for each rate rather than the textbook formula, whose lower limit can go negative (NIST's handbook covers both), and Fisher's exact test when a rule applied on only a few dozen calls (NIST). With 40 rules each checked at 95%, about two will look like they moved even if nothing changed, so confirm a surprise on next week's calls first.

This is the comparison Loops runs on live calls: each rule's miss rate between releases, the same grader version on both sides, and the calls behind any change listed (how it works).

What does a fair comparison look like in practice?

In practice it's one row per rule, with counts on both sides, the change, its interval and a verdict, followed by the calls behind any regression. Example, with illustrative numbers from the example collections SOP used across these posts: release v14 shortened the opening to cut early hang-ups and moved the payment-plan offer earlier. It passed its suite, then ran for two weeks on a 50/50 split with v13 by account, same clients, one grader version. Counts are calls where each rule applied. Each interval is Newcombe's hybrid score interval, which combines the two Wilson intervals.

Newcombe's method treats calls as independent. If accounts took several calls in those two weeks, a cluster bootstrap would widen every interval, but the 2.3 interval would have to grow nearly six times wider to reach zero.

v14 fixed what it set out to fix, and it broke rule 2.3. The stop-request rule doubled on paper, but with about 240 calls a side the interval spans zero, well short of the roughly 750 a side it takes to detect a rise from 3% to 6% with 80% power. Keep collecting to the call count you set for this rule. Here's one of the 190 calls behind the regression.

Agent: Hi, this is Maya, an automated assistant calling for Quillfeather Services on a recorded line. Am I speaking with Luis Ortega?

1.1 met · Opening: business named and recording announced in the first turn

Customer: Who is this? What's this about?

Agent: I can explain once I confirm I'm speaking with Luis. Can you confirm your date of birth?

Customer: July 19, 1990. So what is it?

1.2 met · Right party: verify_identity returned a match at 00:19

Agent: Thanks, Luis. You have a balance of $2,316.40 with Alder Card, and I can set up a payment plan today.

2.4 met · Balance: $2,316.40 matches what get_balance returned at 00:22

2.3 missed · Disclosure: balance stated at 00:24; disclosure not given until 00:41

3.2 met · Payment plan: plan offered before any request for the full balance

Customer: I could do something small on the 3rd.

Agent: Quillfeather Services is a debt collector. This is an attempt to collect a debt, and any information obtained will be used for that purpose. I can set you up for $150 on October 3.

Customer: Okay, the 3rd works.

Agent: You're set for $150 on October 3.

4.1 missed · Promise logged: log_promise called at 00:52 and returned a timeout; no retry

5.1 not checked · Calling hours: needs the call time, the account's address and every number's area code

The suite's disclosure test passed because its caller never asked why the agent was calling, and its criterion checked that the disclosure appeared, not that it came before the debt.

What should you do when a release regresses?

List the calls, decide remediation with compliance, roll back or fix, then prove the next release on live calls.

1. List the calls. Every v14 call that missed rule 2.3, with the quoted line and a timestamp, goes to whoever owns remediation. The list is the work, not the percentage.
2. Decide remediation with compliance. Regulation F is the Consumer Financial Protection Bureau's rule under the Fair Debt Collection Practices Act. It requires the full disclosure in a collector's initial communication with a consumer, and a statement that the communication is from a debt collector in each later one (12 CFR 1006.18(e)). It also requires collectors to keep records that evidence compliance or noncompliance until three years after the last collection activity on a debt, and call recordings for three years (12 CFR 1006.100). Keep the calls, verdicts and release ID together. More in FDCPA and Regulation F rules for AI collection calls.
3. Roll back or fix. All five hosted platforms above can put a previous version back (LiveKit Cloud's instant rollback needs a paid plan). Then fix the order in the prompt, turn the failing live calls into tests (Retell, ElevenLabs, LiveKit and Bland each document a path from a real call to a test), and make the criterion check order, not presence.
4. Prove the next release. Compare v15 with v14 on rule 2.3, confirm the gains on 1.2, 3.2 and 4.1 held, and finish collecting on 4.2. Whether the regression cost money is a separate question, covered in how to measure what a voice agent release earned.

This is general information, not legal advice.

Before your next release ships, stamp the version on every call, pick the five rules you'd least like to break, and write down how many calls and what interval will count as a regression. If you'd like to see that comparison on your own calls first, Loops offers a free 30-day audit.