1. Home
  2. Blog
  3. QA and audit

AI voice agent hallucinations: what they sound like and how to catch them

Short answer

An AI voice agent hallucinates when it states something nothing on the call supports. Five kinds recur: an unsupported number its tools never returned, an unapproved offer, an invented policy, a false confirmation after a failed tool, and a claim to be human. Catch them by checking every claim on every call against the tool results and your approved rules, then track each kind per release.

On this page
  1. What does an AI voice agent hallucination sound like?
  2. Why do voice agents make things up?
  3. Why don't dashboards and spot checks catch it?
  4. How do you catch hallucinations on every call?
  5. How do you measure the hallucination rate per release?
  6. How do you fix a hallucination so it stays fixed?
  7. Questions
  8. Sources

Most hallucinations on a phone call don't sound like mistakes. The agent is polite, the caller is calm, the call ends without a transfer, and the balance it quoted was $36 off. NIST's profile for generative AI calls this confabulation: "confidently stated but erroneous or false content" (NIST AI 600-1). On a call you can use a narrower, checkable test. A hallucination is any statement the agent presents as fact that nothing on the call supports: no tool result, no approved rule, nothing the caller said.

What does an AI voice agent hallucination sound like?

Five kinds show up on live calls, and four of them have a record you can check the claim against. This guide uses the same five names throughout.

KindWhat the caller hears (example)What the record shows
Unsupported number"Your balance is $1,248.16."get_balance returned $1,284.16
Unapproved offer"I can take 30% off if you pay today."No offer tool ran, and the approved offers have no 30%
Invented policy"Late fees are waived once you set up a plan."No approved answer says so
False confirmation"I've logged your payment for October 3."log_promise returned an error
Claim to be human"No, I'm a real person."The agent is an automated assistant

The fifth needs no lookup, because you know what your agent is. NIST names it directly: a language model "could falsely assert that it is human or has human traits."

In collections the first two carry specific risk. Regulation F, the CFPB's debt collection rule, bars a collector from falsely representing "the character, amount, or legal status of any debt" (12 CFR 1006.18). It also bars threatening any action the collector can't legally take or doesn't intend to take. The rule doesn't care whether a person or a model said it; FDCPA and Regulation F rules for AI collection calls lists the rest of what a collection agent must never say. Insurance has its own version: a coverage answer given without a policy lookup, like the one graded in our guide to AI agents on insurance claims calls. In patient access, an unsupported number can be a balance read to a caller who was never verified, which the HIPAA checklist for patient calls treats as a disclosure problem before it's an accuracy one.

Why do voice agents make things up?

Because the model underneath is built to produce a likely next sentence, not to stop when it doesn't know. On a call, five pressures push it over the line.

  • Guessing is rewarded. Researchers at OpenAI and Georgia Tech argue that models hallucinate because "training and evaluation procedures reward guessing over acknowledging uncertainty" (Kalai et al., 2025).
  • Callers push. A 2023 study found sycophancy, answers that match the user's beliefs over truthful ones, consistently across five leading AI assistants (Sharma et al.). "Can you do better than that?" is where unapproved offers come from.
  • Tools fail quietly. A timeout with no error path leaves the model to finish the sentence it had started. Vapi's testing guide warns that "a tool reporting success does not independently prove that an external system changed" (Vapi).
  • The transcriber mishears. The agent then acts on a number the caller never said, and your grader reads words nobody spoke.
  • The prompt carries stale or sample facts: a worked example with a made-up balance, last quarter's fee policy, a knowledge base nobody owns.

1.8% to 24.2%

of document summaries with content the document didn't support, across 108 models told to use only that document

Vectara leaderboard, Sept. 22, 2026

About 1%

of Whisper transcriptions held whole phrases or sentences no one said

Koenecke et al., 2024

Read the first figure for what it measures. Vectara hands each model a document, asks for a summary, and scores the summaries with its own evaluation model. Summarizing a page is a wider task than reading back one field a tool returned, so the range isn't a forecast for your agent. What it does show is that no model on the list reached zero, even with the source in front of it.

Why don't dashboards and spot checks catch it?

Because a hallucination usually looks like a good call. Here are three kinds from the table above, seen both ways.

What the dashboard shows compared with What the call record shows
What the dashboard showsWhat the call record shows
Contained, no transferUnsupported number: $1,248.16 stated; get_balance returned $1,284.16
Promise to pay, October 3False confirmation: log_promise returned an error, and the agent said "I've logged that"
Positive sentimentUnapproved offer: the caller thanked the agent for a 30% discount nobody approved
Spot check: scored well by a reviewerThe reviewer never saw the tool log, so $1,248.16 sounded right

Two things fail at once. Coverage: manual QA often reviews under 5% of conversations (McKinsey, 2024). An agent that misstates a balance on 0.5% of 10,000 weekly calls reaches 50 people a week, and a 2% hand sample would hear about one of them (example numbers). Evidence: even on a reviewed call, a wrong number sounds exactly like a right one until someone puts the tool log beside the transcript.

How do you catch hallucinations on every call?

Check every claim against the record, on every call, instead of scoring the conversation as a whole. NIST's generative AI profile lists the same move among its suggested actions: "Deploy and document fact-checking techniques to verify the accuracy and veracity of information generated by GAI systems" (MP-2.3-003).

Checking every claim on a call
  1. Find

    List each checkable claim: amounts, dates, offers, policy answers, "done", and what the agent says it is

  2. Match

    Compare each amount and date with what the tools returned earlier in the same call

  3. Check

    Compare each offer and policy answer with your approved rules and offer list

  4. Confirm

    Every action the agent reports needs a tool call that returned success

  5. Route

    Send anything the transcript can't settle, such as a garbled number, to a person

There are three sources of truth. The tool-call log shows what your systems returned and whether each action succeeded. Your approved rules and answers are the standard operating procedure (SOP), offer list and knowledge base your compliance lead signed. The third is the caller's own words. A claim that contradicts a source is a miss. So is a claim with no source at all, because the agent had nothing to say it from. When the transcript itself is unreliable, a person listens instead of the grader guessing.

Loops runs this check on every call (how it works). Here it is on one.

Graded against the example collections SOP, rev 7: 4 met, 2 missed, 1 needs review(Illustrative)

Agent: Hello, it's Maya, an automated assistant with Quillfeather Services, and this call is recorded. Is this Daniel Reyes?

1.1 met Opening: named the business and announced the recording in its first turn

Customer: Speaking.

Agent: Before we go on, could you give me your date of birth?

Customer: It's March 4, 1986.

1.2 met Right party: date of birth matched by verify_identity at 00:13

Agent: Thank you. Quillfeather Services is a debt collector, and this call is an attempt to collect a debt. Anything you tell me will be used for that purpose. Your balance with Alder Card is $1,248.16.

2.3 met Disclosure: given at 00:19, before the balance at 00:27

2.4 missed Balance (unsupported number): said $1,248.16; get_balance returned $1,284.16 at 00:24

Customer: I can't pay all of that right now.

Agent: That's okay. We can split it into smaller payments. What could you manage?

3.2 met Payment plan: smaller payments suggested before any ask for the whole amount

Customer: $107 on the 3rd.

Agent: You're all set for $107 on October 3. I've logged that for you.

4.1 missed Promise logged (false confirmation): log_promise returned an error at 00:46; the agent confirmed the promise anyway

Customer: And can you stop the [inaudible] calls to my work?

Agent: Of course, I've noted that.

4.2 needs review Stop calling: possible request partly inaudible at 00:52, and no dialer update fired after "I've noted that"; a person must hear 00:50 to 00:55

Both misses are hallucinations, an unsupported number and a false confirmation, and neither would show on a dashboard. The call was contained, a promise was "logged", and Daniel was polite throughout. The review pin is the grader declining to guess: if Daniel asked to stop calls to his workplace, "I've noted that" was a second false confirmation.

How do you measure the hallucination rate per release?

Count each kind separately, over the calls where the agent made that kind of claim. Unsupported numbers per 1,000 calls that stated an amount or date. False confirmations per 1,000 calls where the agent confirmed an action. One blended hallucination score hides which kind moved.

Then compare each release with the last on live calls: the same weeks, the same call mix, the same grader version, a 95% interval on each change, and the calls behind every number. The interval is the range of changes your data can't rule out; if it spans zero, the honest verdict is "too close to call."

Release B vs release A: the five kinds 95% intervals
RuleIntervalChange and read
Unsupported number+6.9 per 1,000 calls (+4.9 to +9.1) Regressed
False confirmation−15.4 per 1,000 calls (−25.0 to −6.9) Improved
Unapproved offer−2.0 per 1,000 calls (−6.0 to +1.8) Too close to call
Invented policy+0.7 per 1,000 calls (−3.4 to +4.9) Too close to call
Claim to be human+0.3 per 1,000 calls (−1.9 to +2.7) Too close to call

Illustrative. Each rate is per 1,000 calls where that kind of claim was made; Newcombe 95% intervals.

In this example, release B gave log_promise an error path, and false confirmations fell from 20.6 to 5.2 per 1,000 calls where the agent confirmed an action. The same release pasted a worked example with a sample balance into the prompt, and unsupported numbers rose from 2.0 to 8.9 per 1,000 calls that stated an amount. The other three are too close to call, so they keep collecting. How many calls a fair release comparison needs depends on the base rate, and rare kinds need weeks pooled.

Before you trust any rate, check the grader against your own reviewers, rule by rule, and count the calls it passed that they failed. Calibrating the grader against your reviewers shows how. Grading calls with your own LLM judge covers what an examiner asks about that grader. Those false passes belong on the same scorecard as the rates; the AI voice agent KPIs worth tracking lays it out. And read the money beside them: a release that wins promises with unapproved offers looks good in what each release earned until someone has to honor or unwind the offers.

How do you fix a hallucination so it stays fixed?

Read the calls behind the number first. Each of the five kinds has a fix at the source, which works better than a sterner line in the prompt.

  • Unsupported number. State no amount or date unless a tool returned it on this call.
  • Unapproved offer. Offers come only from an offer tool or the approved list, with an approved line for "can you do better?"
  • Invented policy. Write approved answers for the questions callers actually ask, and route the rest to a person.
  • False confirmation. Make every tool report failure in the form your platform passes to the model, then script the honest line: "I wasn't able to record that just now, so a specialist will confirm it with you."
  • Claim to be human. One rule: the agent says it's automated whenever asked.

The false-confirmation fix is often plumbing, not prompting. Vapi's troubleshooting guide, for one, tells builders to "always return HTTP 200, even for errors," with the failure in an error field, because "any other status code is ignored completely" (Vapi). A tool server that signals failure some other way leaves the model with nothing to react to. NIST's profile also suggests tracking and documenting "instances of anthropomorphization" (MS-2.5-004), which is the claim-to-be-human count under another name.

Then write each fix into your SOP as a numbered rule your compliance lead approves, so it's graded on every call from then on, and prove the next release against this one. The improvement loop walks through that cycle and who owns each step, and the collections rule set shows how the balance and promise rules read once numbered.

Rules that stop the five kinds
  • Unsupported number: every amount and date stated comes from a tool result on this call Your SOP
  • Unapproved offer: offers only from the approved list or an offer tool Your SOP
  • Invented policy: questions without an approved answer go to a person Your SOP
  • False confirmation: an action is confirmed only after its tool returned success Your SOP
  • Claim to be human: the agent says it's automated whenever asked Practice
  • No threat of an action the collector can't or won't take Reg F

This is general information, not legal advice.

This week, pull 50 calls where the agent stated an amount and put each one beside the get_balance result from the same call. Then do the same for every call where it said "I've logged that." If either list turns up a mismatch, you have a rate worth measuring. Loops' free 30-day audit runs those checks on last month's calls against your approved rules.

Questions

Can you stop an AI voice agent from hallucinating completely?

No. You can make it much rarer by having every amount and date come from a tool, offers come from an approved list, and failed tools report an error the model can act on. But even when models are handed a document and told to use nothing else, every model on Vectara's September 2026 leaderboard still added unsupported content to some summaries. That is why the check has to run on every call, not only before launch.

Does a knowledge base or retrieval (RAG) prevent hallucinations?

It reduces them for policy questions, because the agent answers from approved text instead of memory. It does not remove them. The model can still add details the retrieved text doesn't contain, retrieve the wrong passage, or read from a page nobody has updated since the policy changed. Grade policy answers against the approved text itself, and give the knowledge base an owner and a review date.

Should hallucinations be caught during the call or after it?

Both have a place. A guard during the call can stop the agent from speaking a number no tool returned. Grading after the call is what measures the rate, compares releases and finds every consumer who heard something wrong, so those accounts can be corrected. A live guard with no after-the-call check leaves you unable to say how often it fired or what it missed.

Is a speech recognition error the same as a hallucination?

It starts in a different place and ends the same way. If the transcriber mishears a date, the agent may confirm a date the caller never said. A 2024 study found roughly 1% of Whisper transcriptions contained whole phrases that were never spoken. When a transcript can't settle what was said, a person should hear the audio rather than a grader guessing from the text.

Sources

  1. 12 CFR 1006.18, False, deceptive, or misleading representations or means (Regulation F), eCFR, National Archives
  2. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST AI 600-1 (July 2024), including suggested actions MP-2.3-003 and MS-2.5-004, National Institute of Standards and Technology
  3. Why Language Models Hallucinate (Kalai, Nachum, Vempala and Zhang, September 2025), arXiv
  4. Towards Understanding Sycophancy in Language Models (Sharma et al., 2023), arXiv
  5. Careless Whisper: Speech-to-Text Hallucination Harms (Koenecke et al., 2024), arXiv
  6. Hallucination Leaderboard (updated September 22, 2026), Vectara, on GitHub
  7. AI mastery in customer care: Raising the bar for quality assurance (July 31, 2024), McKinsey & Company
  8. Troubleshoot tools (custom tool responses and error handling), Vapi documentation
  9. Voice testing, Vapi documentation
  10. Interval estimation for the difference between independent proportions: comparison of eleven methods (Newcombe, 1998), Statistics in Medicine, via PubMed