AI voice agent: how a call works, where it fails and which rules apply

How an AI voice agent takes a call from the first word to the last

An AI voice agent is software that holds a spoken phone conversation in place of a person. The caller dials an ordinary number, and a telephony service passes the audio to the system. From there the call goes through four steps that repeat for every sentence.

First, speech recognition turns the caller’s words into text; Wikipedia describes it as technology that converts spoken language into text. Second, a language model reads that text together with your script and your facts, and writes the next reply. Third, speech synthesis, which Wikipedia defines as the artificial production of human speech, reads the reply aloud. Fourth, the telephone line plays it back and listens again. Around them sit a calendar to book a slot, a customer list to save the result and a rule that passes the call to a person.

Jobs that suit a voice agent: qualifying, booking and reminding

Inbound means the customer calls you. The agent can answer at any hour, ask the same short questions every time and write down the answers: what the person needs, the budget, the city, the preferred day. A salesperson then picks up a prepared note instead of a blank sheet. Qualification is the most requested job: three or four questions decide whether a lead deserves a salesperson’s hour, and the meeting can go into a calendar on the spot.

Outbound means the agent calls. Typical jobs are a first call to someone who left a request on your site, a reminder about tomorrow’s appointment, a confirmation of a delivery slot, or a short survey after a purchase. What these share is that the person expects the call or has asked for it. Cold calling strangers is a different matter, and the rules further down apply to it with full force.

Where voice agents fail and when a person must step in

Wikipedia’s article on speech recognition lists what makes the first step hard: differences in accent, dialect, pronunciation and speaking speed, and background noise such as a car or a busy street. The same article says that continuous, spontaneous speech with hesitations like “uh” and false starts is the hardest style to recognise. That is how real customers talk on the phone.

Accuracy has a standard measure, the word error rate. It counts wrong, missing and extra words in the transcript and divides the total by the number of words actually spoken, so a lower score is better. Speech synthesis has its own limits: Wikipedia notes that rhythm and emotional expression are still active research problems, so a correct reply can still sound flat.

Delay is the failure that callers notice first. When the pause between the caller’s sentence and the agent’s reply is long, people talk over it or hang up. Plan the handover before launch: the agent passes the call to a person when the caller asks for one, when the same question fails twice, when a complaint or anger appears, or when money or a contract is discussed. The handover should carry the notes.

The rules for AI voices on the phone: the FCC, the FTC and the EU

In the United States the Federal Communications Commission issued a Declaratory Ruling about exactly this technology, adopted on 2 February 2024 and released on 8 February 2024. It confirms that the Telephone Consumer Protection Act’s limits on an “artificial or prerecorded voice” cover current AI technologies that generate human voices. Calls that use them need the prior express consent of the called party, unless there is an emergency purpose or an exemption. For a call that introduces an advertisement or is telemarketing, the FCC’s rules require prior express written consent.

The ruling adds that such a message must state, at the beginning, who is responsible for the call, and that telemarketing calls must offer a way to ask to stop. Separately, the Federal Trade Commission’s guide to its Telemarketing Sales Rule says sellers and telemarketers must not call numbers on the National Do Not Call Registry or on the seller’s own do-not-call list. Which rule covers your case is a question for a lawyer, not for an article.

Elsewhere the rules differ by country and change. On its page about the AI Act, the European Commission says people should be told when they are dealing with a machine, and that the transparency rules come into effect in August 2026. Article 50(1) of the Act asks providers to design systems that talk directly to people so that those people are informed they are interacting with an AI system, unless that is already evident to a reasonably observant person. A greeting that says the caller is speaking with an automated assistant is the safe default.

What the agent needs from you: a script, answers and careful handling of recordings

A voice agent knows only what you give it. That means a call script with the opening line, the questions and the point where it stops; a list of real customer questions with approved answers; opening hours, prices you are willing to say aloud, and the calendar or list it may write into. Everything else gets an honest sentence that a person will call back.

Recordings and transcripts are personal data in many places. Decide before launch whether calls are recorded, who can listen, how long they are kept and how a customer can ask for deletion. Which rules apply to you is again a question for a specialist. Collect only what the next step needs: a phone number and a time are enough to book, a passport number is not.

Measuring a voice agent: answer rate, booked meetings and handover rate

Three numbers tell most of the story. The answer rate shows how many calls reached a real conversation. The booked-meeting rate shows how many conversations ended in a slot in the calendar. The handover rate shows how often a person had to take over, and a handover is not a failure if it happened for a good reason.

Read the numbers together with a sample of recordings, because a figure alone hides what went wrong. No result can be promised in advance; it depends on your offer, your list and your script. The aim of measuring is to find the next sentence to fix.

A test plan to run before the first real call

Start with calls made by your own team, with a written list of cases: a clear request, a vague one, a caller who interrupts, a caller who asks for a person, a caller who asks something the script does not cover, and a caller who says stop. Add accents and background noise on purpose, since Wikipedia names both as hard cases. Write down every answer that sounds wrong, and fix the script before the voice.

Then run a small pilot on a short list of people who have agreed to be called, and keep a person on the handover line. Check that every booking lands in the calendar with the right name and time, and that a caller who says stop is never called again. Widen the list only after a week of calls you would be content to play to a customer.

Build it yourself or order it: what to bring to a voice agent project

Building alone is realistic if you already have a telephony account, a developer and time to test. The first working demo is the easy half; the harder half is the script, the handover, the consent wording and the weekly listening. If nobody on your side will own those, ordering the work is simpler.

Bring four things: the offer in plain sentences, the questions the agent should ask, the list of answers it may give, and a person who will take the handovers. Add a clear view of who will be called and on what basis, since consent is a matter for you and your lawyer.

VITON13 Studio builds an AI voice sales agent as a package, and shows a free draft first: a test call on your offer. The fixed price is $313, monthly support is $53 a month, and the build takes 10 to 18 working days. Telephony minutes and model usage are paid separately. A receptionist for incoming calls and chats is a related package at $213 plus $53 a month.