Voice Agents in Production

TL;DR

• Voice quality is a system property. Turn-taking, latency, transcription, reasoning, speech, integrations, and recovery all affect the caller's experience.
• Use explicit conversation states for repeatable work. Allow flexible language inside a state, but keep transitions, permissions, and handoff rules testable.
• Evaluate task completion and recovery with real calls. A polished synthetic voice cannot compensate for an incorrect booking or a broken transfer.
• Model the full cost per resolved call, including telephony, models, retries, human review, and escalations. Recalculate it from current usage and pricing.

---

Voice is useful when speaking is already the natural interface: answering an inbound call, booking an appointment, routing an enquiry, or collecting information while someone is mobile. It is not automatically better than a form, message, or human conversation.

The product decision starts with the job. Choose voice when it reduces effort for the user and the workflow can be completed or handed off safely. Do not choose it because a natural-sounding demo is impressive. The broader AI UX design framework helps compare voice with inline assistance, workspaces, and delegated agents.

Treat latency as a turn-taking budget

Callers experience latency as silence. The system should respond quickly enough that people understand whether it heard them, is working, or needs clarification. There is no universal threshold because languages, networks, tasks, and caller expectations vary.

Measure the complete turn:

Turn latency = Endpoint detection + Transcription + Reasoning + Tool calls + Speech generation + Network delay

Track median and tail latency separately. The median describes the common experience; the tail reveals the pauses that cause interruptions, repeated questions, and abandoned calls.

Useful design responses include:

• Stream transcription and speech where the stack supports it.
• Keep routine responses short enough to begin quickly.
• Acknowledge a tool call when silence would otherwise be confusing.
• Cache stable business information only when freshness and privacy rules permit it.
• Give slow integrations their own timeout, retry, and fallback behaviour.

A filler phrase is not a latency fix. Use one only when it truthfully explains that work is happening. Repeated or unnecessary filler makes the agent sound evasive.

Design a state machine, not a script recital

Open conversation needs flexibility, but repeatable business work needs structure. Define a small set of states with a goal, required information, allowed actions, exit conditions, and recovery path.

A booking workflow might use:

The model can interpret natural language within a state. Code and policy should constrain high-consequence transitions such as committing an appointment, taking payment, changing an account, or disclosing personal information.

One question at a time is a useful default. Confirm information where an error has a material consequence, not after every harmless phrase. The aim is a clear conversation, not a mechanical interrogation.

Make recovery a first-class flow

Design the expected failure paths before launch:

• The caller corrects the agent.
• Speech is ambiguous or the transcript is incomplete.
• The requested option is unavailable.
• The caller changes intent midway through the call.
• A business system is slow or unavailable.
• The request is outside policy or the agent's permissions.
• The caller asks for a person.

For each path, specify what the agent retains, what it asks again, what it must not infer, and when it hands off. A recovery that preserves confirmed details is less frustrating than restarting the interaction.

A handoff is a product outcome

A successful handoff preserves continuity. Transfer three things together:

1. The live interaction, through the appropriate transfer mechanism.
2. A structured summary of the caller's intent, confirmed details, and reason for escalation.
3. The transcript or relevant evidence, subject to consent, privacy, and retention policy.

Test the receiving experience, not just the agent's side. The human should know why the call arrived and what has already been attempted. The caller should not need to repeat information the business has safely captured.

Human availability is part of the feature. If no one can receive the transfer, the agent needs an honest alternative such as a callback request, message capture, or safe return to a supported self-service path.

Build evals from real conversations

Synthetic conversations are useful for coverage, but they do not reproduce every accent, interruption, background noise pattern, emotional cue, or unexpected turn. Combine them with consented and appropriately handled production evidence.

Evaluate at four levels:

Segment the results. Aggregate completion can hide failures for a language, accent, noisy environment, intent, or integration path. Review the worst important segment, not only the average.

Use human review where judgement is required, deterministic checks for structured outcomes, and adversarial tests for abuse and policy boundaries. Add recurring production failures to the regression set. The AI evaluation framework covers test-set design, graders, thresholds, and drift.

Model cost per resolved call

Quoted per-minute or per-token prices are only inputs. The product unit is a resolved call or another customer outcome.

Cost per resolved call = (Telephony + Transcription + Model + Speech + Tooling + Retry + Human review + Escalation cost) ÷ Resolved calls

Compare this with the current cost and quality of the whole workflow, including missed demand, callbacks, rework, and human coverage. Do not assume automation is cheaper. A low inference bill can hide expensive escalations or failed outcomes.

Stress-test the model with observed call duration, concurrency, retry rate, escalation rate, and current provider prices. Recalculate after material model, prompt, voice, or integration changes. Use the complete AI cost-per-task model, not the advertised voice price alone.

Choose the launch boundary

Good first workflows tend to be frequent, narrow, reversible, and supported by reliable systems of record. Poor first workflows involve emotional conflict, broad discretion, missing data access, or consequences the agent cannot reverse.

Launch with a written boundary:

• Supported intents and caller segments
• Allowed tools and actions
• Information the agent may collect or disclose
• Confirmation requirements
• Escalation and fallback conditions
• Quality, risk, latency, and cost thresholds
• Owner for incidents and operational review

The agent ownership contract turns these boundaries into named responsibilities for context, permissions, evals, incidents, and retirement.

Verify disclosure, consent, recording, privacy, accessibility, and sector-specific obligations for every jurisdiction in which the feature operates. Legal requirements and caller expectations can differ.

Production readiness review

• Real-call testing covers the main intents, important segments, interruptions, and background noise.
• Tail latency is measured across the full turn, including slow tools.
• Material details are confirmed before consequential actions.
• Permissions follow least privilege and sensitive data has an explicit handling policy.
• Integration failures have timeouts, retries, and safe fallbacks.
• Human handoff works end to end, including when no human is immediately available.
• Structured outcomes are verified against the source of truth.
• Risk failures and recurring production errors enter the regression suite.
• Cost is measured per resolved outcome, including escalation and rework.
• A named owner reviews quality, incidents, and scope changes.

The anti-pattern: optimising the voice

The team spends weeks tuning warmth, personality, and pronunciation. The agent sounds excellent in a demo. In production it books the wrong service, loses details during transfer, and has no safe response when the scheduling system fails.

Voice quality matters, but it is one layer of the product. Start with correct outcomes, recoverable failures, clear boundaries, and a humane handoff. Then improve how the system sounds.