How to Test an AI Voice Agent Before It Takes a Real Call: A 30-Call Script, the Built-In Simulators and What to Measure
A vet clinic's first AI receptionist handled appointments correctly about 60 percent of the time in testing. Here is how to find your number before your customers do: a scripted test plan, the simulation tools inside Retell, Vapi and ElevenLabs, two outside testing services, and what it all costs.
The 60 percent receptionist
A Voice AI Receptionist for Veterinary Clinics That Books Appointments Automatically, documented on this site, reports that its first version handled appointments correctly about 60 percent of the time during initial testing. The known problems were the ones every builder eventually meets: it misheard names and phone numbers, and it offered appointment slots that did not exist. The caller feedback on how it sounded was positive. That is the trap. A voice agent that sounds good and books the wrong thing is worse than voicemail, because nobody notices until a customer turns up on the wrong day.
Most of the build guides on this site, including our own, end with a line like call it twenty times and fix what you find. That is the start of testing, not the end. This guide is the rest: what breaks, the four layers of testing in order of cost, a 30-call script you can run with two phones, the simulation features now built into Retell, Vapi and ElevenLabs, what the outside testing services charge, and the bar to clear before the agent takes a real call.
What goes wrong, in the order you will meet it
Name and number capture fails first. The vet clinic listing names it, and so does nearly every agency on this site that sells receptionists. Phone numbers read at speed, names with unusual spellings and callers who talk over the agent produce wrong contact details, which turns a booked job into a no-show.
Invented availability comes second. When the calendar tool times out, returns nothing or returns a conflict the prompt did not plan for, the model fills the gap with a plausible time. The vet clinic listing calls these hallucinated appointment slots. An Emergency Plumbing AI Voice Agent Built in Three Steps With Vapi and n8n and AI Voice Receptionist With Calendar Booking Using Vapi and n8n both passed their live demos because the calendar lookup worked. You have to test what happens when it does not.
Scope creep is third. The agent quotes a price, promises a same-day visit, or gives medical or legal advice because a caller pushed and the prompt did not forbid it. Then come transfer failures, where the line drops, and the quiet ones: long silences while a tool runs, talking over the caller, or claiming to be a person when asked, which is a legal problem in some places and a trust problem everywhere.
The one agency build on this site that describes testing in detail, AI Voice Receptionist for HVAC Businesses: Full Build From Scratch Using Retell AI, used a simulated-testing platform to generate test cases and run AI personas against the agent before launch, and says its production systems have been through 100 to 200 live calls per deployment. One AI Agent Handled Every Phone Call at an HVAC Company for 30 Days Straight treated the first month itself as the test. Both point the same way: nobody serious ships after twenty calls.
The four layers of testing
The cheapest tests find the most bugs, so run them in this order: text simulations inside your platform, scripted calls by your own people, an outside testing service if the stakes justify it, and a soft launch that only takes after-hours calls while you read every transcript.
Text simulations are now built into the three platforms behind most of this site's receptionists. Retell's simulation testing has an AI-simulated user run a scenario you describe as a text conversation, grades it against success criteria you write, and lets you save cases and re-run them as a regression suite. Retell's docs say it runs as text, does not support agents on a custom LLM, times out after 10 minutes, and bills per message. Vapi offers Evals, which feed the agent a conversation up to a decision point and check its next response by exact match, pattern or an AI judge, and Simulations, where an AI tester with a goal and facts holds a full conversation, in chat or voice mode; Vapi's docs say chat simulations are faster and cheaper than voice ones. ElevenLabs' agent platform, now called ElevenAgents, lists automated tests and simulations alongside conversation analysis.
Scripted calls by humans catch what text cannot: mishearing, noise, interruptions and latency. Outside services such as Cekura and Hamming run hundreds of simulated voice calls, test concurrency and probe for prompt injection, which matters more for an agent that can transfer calls or read account data. The soft launch catches the rest.
A worked example: a two-chair dental office going live
Picture a dental office with two dentists, a front desk of two, and about 250 calls a week. They built an agent to answer after hours and overflow, book cleanings, and route emergencies, the pattern described in A Dental Clinic AI Receptionist That Cut Front Desk Workload by 35%, which reports missed appointments down 28 percent for one clinic, a self-reported figure.
Week one, the office manager writes 50 text test cases: 20 routine bookings with different names and insurance questions, 10 emergencies, 10 out-of-scope requests such as prescription refills and price negotiation, and 10 tool failures where the schedule is full or the booking system is down. They run in the platform's simulator in an afternoon and fail on 11. Most failures are the agent answering clinical questions it should decline. The prompt gets a hard rule and the 11 cases are re-run until they pass.
Week two, the two front-desk staff make 30 scripted calls from their cars and the break room, scoring each on a Google Sheet. Four calls capture a wrong digit in the callback number. The fix is a spell-back step: the agent reads the number back and waits for a yes. Week three, the agent takes only calls after 6pm. The manager reads every transcript each morning for two weeks and finds two more problems nobody scripted, including a caller who switched to Spanish. Only then does it take daytime overflow. Total cost of testing was under $40 in call minutes and about six hours of staff time. This is an illustration of the method, not a documented client.
Step by step: test it properly in three weeks
Step 1: Write the pass criteria before the first test
For each thing the agent does, write what correct means in one line. Correct booking means the caller's name is spelled back and confirmed, the number is read back and confirmed, the slot exists in the real calendar, the event is created with the reason for the visit, and the confirmation text or email goes out. Correct emergency handling means the agent transfers or takes a callback within 60 seconds and never offers a time. Correct out-of-scope handling means the agent declines, offers a callback, and does not guess. If you cannot write the line, you cannot grade the test.
Step 2: Build the 30-call script
Write 30 short caller cards. Ten are normal bookings with varied names, including two that are hard to spell and two read fast. Five are emergencies, including one at 2am and one where the caller is angry. Five are out of scope: a price negotiation, a medical or legal question, a job you do not do, a sales call, and someone asking whether they are talking to a robot. Five are difficult callers: constant interruptions, a noisy truck cab, a long pause, a caller who changes their mind twice, and one who speaks a second language. Five are tool failures, covered in Step 5. Put them in a Google Sheet with columns for the caller card, what should happen, what happened, and pass or fail.
Step 3: Run the text simulations first
In Retell, open the agent's testing section, create a test case with a user prompt describing the caller, write two to four success criteria that match Step 1, and run it; every criterion must pass for the case to pass, and the saved cases become your regression suite. In Vapi, create Evals for the decision points, such as whether the agent transfers an emergency, and Simulations for full calls, starting in chat mode because it is cheaper. In ElevenAgents, use the testing feature to define the same cases. Convert all 30 cards plus 20 variations into text cases. Fix the prompt until they pass, and keep the cases.
Step 4: Make the 30 calls with real phones
Two people, two phones, half a day. Call from a car with the window down, from a kitchen with a dishwasher running, and from a quiet room. Score every call on the sheet against Step 1 right after hanging up, then read the platform's transcript and compare it to what the caller actually said. Transcript errors are transcription errors, which you fix with a different transcriber setting or a spell-back step, not with prompt changes.
Step 5: Break the tools on purpose
Disconnect the calendar integration and call. Fill the calendar completely and call. Point the booking webhook at a dead URL and call. Make the CRM lookup return two matches for the same name. In each case the right answer is some version of I cannot see the schedule right now, I will have someone call you back within the hour, followed by a real callback task for a person. Any invented time, price or confirmation is a fail, and it is the most expensive kind.
Step 6: Fix, then re-run everything, every time
A prompt change that fixes the emergency case can quietly break the booking case. After every change, re-run the saved text cases, and after any change to tools, voices or transcriber settings, re-run the phone script too. Retell and Vapi both keep saved cases for exactly this reason. Builders on this site describe three or four rounds before an agent is stable; budget for that.
Step 7: Soft launch on after-hours calls only
Forward the line to the agent after closing time and on weekends, keep the daytime line with your staff, and read every transcript each morning for 14 days. Vendors sell call-scoring add-ons, covered under costs below, but at a small business's volume a person reading transcripts is faster and cheaper. Keep a list of anything a real caller did that the script never imagined, and turn each one into a new test case.
Step 8: Set the bar and the kill switch
Decide the go-live numbers in advance. A reasonable bar for a booking agent is at least 95 percent correct capture of name and number across the last 30 scored calls, zero invented availability, and every emergency handled correctly. Decide who can pull the plug and how: usually one person who can flip the call forwarding back to the office in under a minute. Write the number to call to do it on a sticky note by the front desk.
What this costs
Testing spends the same minutes a real call does, plus any outside service. As of November 2026, Retell's pricing page lists pay-as-you-go voice agents at $0.07 to $0.31 a minute depending on model and voice, with $10 in free credits and the first 100 minutes of its AI quality-assurance scoring free. Vapi charges $0.05 a minute for hosting plus model, voice and transcription at cost; its own calculator estimates $82 to $129 a month for 1,000 minutes, and new accounts get $5 in free credits. ElevenAgents charges $0.08 a minute for call minutes with the language model billed at cost on top, and its free plan includes 15 minutes. Thirty scripted calls at about three minutes each is roughly 90 minutes, so the phone script costs $10 to $30 on any of the three, and the text simulations cost cents.
Cekura's pricing page lists pay-as-you-go voice testing at $0.25 a minute with 300 free credits, which it says is about 60 minutes, and a Startup plan at $500 a month with about 2,000 minutes of voice testing. Hamming does not publish prices; its pricing page says it is quoted per deployment. For a single-location business, the free credits on Cekura are worth using once, and the rest of the budget is better spent on staff time: plan on six to eight hours across three weeks.
When not to bother, and when to stop
If you use a managed receptionist product where the vendor runs the testing, do the phone script and the soft launch and skip the rest. If you cannot write the pass criteria in Step 1, do not launch; you have not decided what the agent is for. And if accuracy is still under 90 percent after three full rounds of fixes, the problem is usually the platform, the transcriber or the scope, not the prompt. Narrow the job to answer, capture and transfer, or compare platforms using this site's Retell vs Vapi guide, rather than adding a fourth page of instructions.
Frequently Asked Questions
How do you test an AI voice agent?▾
In four layers: text simulations inside the platform to check logic and tool use, scripted phone calls by your own staff to check hearing and handling, optional outside testing services for scale and security, and a soft launch on after-hours calls while you read every transcript for two weeks. Re-run the saved tests after every change.
How many test calls does an AI receptionist need before going live?▾
Plan on 50 text simulations and 30 scripted phone calls before a soft launch, then a two-week after-hours period with every transcript read. An HVAC agency documented on this site reports 100 to 200 live calls per deployment before it considers a system proven, a self-reported figure.
Do Retell, Vapi and ElevenLabs have built-in testing?▾
Yes. Retell has simulation testing with an AI-simulated user, written success criteria and a regression suite, run as text and billed per message. Vapi has Evals for single decisions and Simulations for full conversations in chat or voice mode. ElevenAgents lists automated tests and simulations in its platform.
What does voice agent testing cost?▾
Scripted calls cost ordinary call minutes: roughly $10 to $30 for 30 three-minute calls on Retell, Vapi or ElevenAgents as of November 2026. Cekura lists voice testing at $0.25 a minute with about 60 free minutes, and a $500 a month Startup plan. Hamming quotes per deployment.
What is the most common AI voice agent failure?▾
Wrong names and phone numbers, followed by invented appointment times when the calendar tool fails. Fix the first with a read-back and confirm step and the second with an explicit rule that the agent must offer a callback, never a time, when it cannot see the schedule.