Voice AI Agents in Production: Enterprise Architecture for Real-Time Voice-First Systems

Why Voice Agents Are Different

Every other agent architecture we build has one property in common: latency does not have to be human. A chatbot that takes 8 seconds to respond is a slow chatbot; a document-extraction agent that takes 30 seconds is fine. Voice agents lose their reason for existing at 800 ms of end-to-end response latency. Above that number, people start talking over the agent, and the conversation breaks down.

That one number cascades through every architectural decision. You cannot spend 400 ms on a network round-trip to a slow reasoning model, then 300 ms on text-to-speech, then 200 ms on your telephony provider's buffer, and then hope. Every hop has a budget, every budget is small, and every budget is enforced by the physics of human conversation, not by an SLA you can renegotiate.

A production voice agent is therefore not a chat agent with a microphone bolted on. It is a real-time system with an LLM as one component, and everything else — turn-taking, interruption handling, background noise, telephony jitter — is engineering work that has nothing to do with prompting.

The Five-Layer Reference Architecture

The mental model we deploy has five layers, and understanding what belongs at each is most of the battle.

Telephony ingress. How the audio enters your system. For phone-based agents this is a SIP trunk from a provider (Twilio, Telnyx, Vonage, or an on-prem PBX). For web/mobile agents it is a WebRTC session, usually managed by a real-time transport service. This layer's job is to hand you an audio stream and take one back — nothing more.

Speech-to-Text (STT). Streaming transcription of the caller's audio into text, with word-level timestamps. Latency below 300 ms per utterance is the bar. Deepgram, AssemblyAI, and Speechmatics dominate the commercial market; OpenAI Whisper (fine-tuned) and NVIDIA Parakeet are the self-hostable options. Streaming (partial transcripts as words arrive) is table stakes; if your STT only returns full transcripts, your agent will feel a full turn slower than the competition.

Reasoning + tools. The LLM that decides what to say and when to invoke a tool (check calendar, look up order, escalate to human). This is the layer you already know how to build if you have built any other agent. For voice it needs one extra quality: streaming responses, so the TTS layer can start speaking before the model has finished generating.

Text-to-Speech (TTS). Streaming synthesis of the model's response back into audio. ElevenLabs, Cartesia, and Rime AI are the current commercial leaders on latency-plus-quality; Kokoro and XTTS are the self-hostable options. Streaming is again the bar — sub-300 ms time-to-first-audio-byte for a natural conversation.

Turn-taking and interruption control. This is what makes the difference between an agent that feels like a conversation and one that feels like walkie-talkies. Voice Activity Detection (VAD) decides when the caller has finished speaking; end-of-turn detection decides when to stop the agent's speech if the caller interrupts. Silero VAD is the default choice; the newer 'turn detection' models (LiveKit's turn-detection-v2, or the built-in end-pointing in the Realtime APIs) are what push the experience from acceptable to natural.

The Speech-In / Speech-Out Alternative

The Voice Orchestrator Layer

Between telephony and the LLM sits the piece that ties everything together: the voice orchestrator. This layer owns the real-time session, dispatches audio to STT, tokens to the LLM, and audio back to telephony; runs the VAD; and handles interruptions. It is the one component you should not try to write from scratch on your first voice project.

Retell AI and Vapi are the leading managed platforms — you point them at your LLM (any provider, or a gateway), give them a system prompt and a set of function-callable tools, and they handle the pipeline. Both integrate with Twilio and Vonage for telephony out of the box, and both expose webhooks for post-call processing. Retell tends to have the edge on turn-taking naturalness in 2026; Vapi tends to have the edge on customisation and multi-language.

LiveKit Agents is the open-source alternative — a real-time media server plus a Python agents framework. It runs your own infrastructure, is provider-agnostic across STT / LLM / TTS choices, and integrates directly with LiveKit's WebRTC transport. If you already run LiveKit for video, or if regulatory requirements demand on-prem media, this is the natural fit.

Pipecat is the framework layer beneath many of the above — Python primitives for composing real-time voice pipelines, used by teams building their own orchestrator. Choose this when Retell or Vapi cannot accommodate a specific requirement (custom VAD, unusual telephony provider, exotic language support).

For a first enterprise deployment the pattern that almost always works is: Retell or Vapi as the orchestrator; your existing LLM gateway as the LLM; your existing observability platform receiving traces via webhook. The orchestrator becomes a boundary, not a lock-in.

Tools, Function Calling, and Backends

The tool interface a voice agent needs is the same as any other agent — a set of typed functions the model can invoke — but with two additional constraints.

First, tool calls must be *fast*. A caller will not wait five seconds while your CRM API responds. Budget tools at 300-500 ms for the happy path; anything slower needs an async pattern where the agent says something like 'let me pull that up' while the tool runs in the background. This is not optional theatre — it is what preserves the conversation's rhythm.

Second, tool selection must be *cheap* to explain. In text agents the model can be verbose about why it chose a tool; in voice, the audible turn is the reasoning. Prefer few well-named tools over many finely-scoped ones. If the agent is picking between twenty tools per turn, the latency will suffer and the reasoning will be hard to audit.

The backend integration pattern that ages well: expose backend operations as MCP servers or a small internal API, and let the orchestrator's function-calling layer bind to them. Retell, Vapi, and LiveKit Agents all speak this pattern. Never let the voice agent talk directly to systems of record — always through a service layer that enforces auth, tenant isolation, and audit logging. Voice sessions are ephemeral; the audit trail must not be.

A Production-Shaped Voice Agent Configuration

# retell-agent.yaml — a Retell configuration wired to enterprise infrastructure

agent_name: support_triage_voice
voice_id: 11labs-adrian     # ElevenLabs voice, sub-300ms TTFB

# ---- Model routed through the enterprise AI gateway ----
response_engine:
  type: custom_llm
  llm_websocket_url: wss://gateway.internal.example.com/voice/v1/agent/support
  # gateway handles: key mgmt, budget, provider fallback (Claude -> GPT),
  # cost accounting per tenant, prompt versioning

# ---- Telephony ----
telephony:
  provider: twilio
  numbers: ['+1-555-01234']
  webhook_url: https://api.example.com/voice/inbound

# ---- Turn-taking calibrated for enterprise callers ----
interruption_sensitivity: 0.6      # allow the agent to be interrupted mid-sentence
end_call_after_silence_ms: 4000
backchannel_frequency: 0.3         # 'mm-hmm' at natural moments
responsiveness: 1.0                # start speaking as soon as the model streams

# ---- Tools available to the agent ----
functions:
  - name: lookup_order
    description: 'Look up an order by ID. Read-only.'
    url: https://api.example.com/tools/orders/lookup
    speak_during_execution: 'Let me pull that up for you.'

  - name: propose_refund
    description: 'Propose a refund up to $200. Requires supervisor approval before it is executed.'
    url: https://api.example.com/tools/refunds/propose
    speak_during_execution: 'One moment while I check what we can do.'

  - name: transfer_to_human
    description: 'Transfer the call to a human agent when the caller is upset or the case is out of scope.'
    url: https://api.example.com/tools/telephony/transfer

# ---- Post-call ----
post_call_analysis:
  webhook_url: https://api.example.com/voice/post-call
  # We forward the transcript + tool trace to Langfuse from this webhook,
  # then run a scored evaluation against the golden set nightly.

data_retention:
  transcripts_days: 90              # regulated workloads may cap this at 30
  recordings_days: 30
  pii_redaction: on                 # Retell can redact PII before storing

Latency Engineering: Where the Milliseconds Go

Enterprise voice quality is decided in the latency budget, and the budget is unforgiving. A rough breakdown of where 800 ms of end-to-end latency spends itself in a healthy pipeline: 100 ms of telephony transport (SIP jitter buffer, WebRTC packetisation), 200 ms of STT (streaming, so this is time from utterance-end to final transcript), 200 ms of LLM time-to-first-token (streaming), 200 ms of TTS time-to-first-audio-byte, 100 ms of VAD + turn-taking overhead.

The first optimisation most teams reach for is a faster LLM, and it is usually the right one — a small model streaming from the same region as your orchestrator will save 300 ms compared to a large model called via a distant region. The second is co-location: run the orchestrator, the LLM (if self-hosted), and the STT / TTS in the same cloud region as your telephony provider's media servers. A 40 ms cross-region hop, multiplied by every turn, is what makes the difference between 'natural' and 'a bit slow.' The third is streaming end-to-end: any component that returns a full response only after producing all of it is quietly killing your latency budget.

The pattern we run for critical paths: LLM behind vLLM in the same region as the orchestrator, Deepgram or fine-tuned Whisper for STT, Cartesia for TTS (competitive latency and quality), and a health-check webhook every 30 seconds that pages if any component's p95 latency crosses its budget.

Observability, Evaluation, and the Reality of Voice Failures

Voice failures are qualitatively different from text failures. In a text agent the failure mode is a bad answer; in a voice agent the failure mode is often a bad *conversation* — the agent talked over the caller, missed an interruption, mispronounced the customer's name, misinterpreted a heavily accented word, or produced audio that sounded correct but did not land.

The observability stack that works has three layers. Structured traces per turn — STT confidence, LLM prompt / output, tool calls, TTS voice ID, timings for every stage — forwarded to your Langfuse or LangSmith instance from the orchestrator webhook. Post-call analysis — a slower job (LLM-as-judge, or a fine-tuned classifier) that scores the whole conversation against a rubric (task success, sentiment shift, containment rate) and files the score in your dashboard. Sampled human review — every dashboard we build for a voice agent has a 'listen to 20 random calls this week' widget for the operations owner, because there is no substitute for listening.

The evaluation metrics that actually track quality: containment (percentage of calls resolved without human transfer), latency p95 by turn, interruption recovery rate (did the agent recognise and adapt to the interruption), tool-call success rate, and post-call CSAT if you can collect it. Do not deploy a voice agent without at least the first three.

Where Voice AI Agents Tend To Be the Right Choice

Where It Is Not the Right Choice

Voice is a terrible medium for anything that needs a form, a document, or a table — the caller has no place to see it. Complex authentication flows, most B2B configuration work, and anything requiring the caller to compare options are all better served by chat or a screen. Voice agents also do not solve for emotionally difficult conversations — cancellations, complaints, bereavement services — where the point of the call is human contact, not throughput. And in regulated flows where every word must be exact (financial advice, medical diagnosis, legal advice), the risk profile of a generative voice agent is currently unacceptable without a human in the loop; the agent's role there is intake and routing, not decision-making.

Seven Decisions Worth Making Before Your First Deploy

• Pipeline versus Realtime. Decide based on latency requirements and per-minute economics. Pipeline for volume, Realtime for VIP, hybrid for both. Do not default to Realtime just because it is newest.
• Managed orchestrator versus self-hosted. Retell or Vapi for the first deployment. LiveKit Agents or Pipecat only if data residency, custom telephony, or exotic language support demands it.
• LLM routed through your gateway. Voice agents should not have their own model contracts. Point the orchestrator at your existing AI gateway so budgets, keys, and provider fallbacks apply here as everywhere else.
• Consent and recording policy. Decide with legal and compliance *before* build. Retention periods, PII redaction, opt-out flow — these are launch blockers if not resolved.
• Fewer, larger tools. Voice does not tolerate long tool-selection reasoning. Aim for four to six tools per agent, each covering a coherent capability.
• Post-call evaluation as a build step. Nightly LLM-as-judge scoring against a rubric, plus weekly human review of a sample. Voice quality drifts silently without this.
• Latency SLOs per component. Publish and alert on them: STT p95 300 ms, LLM TTFT 400 ms, TTS TTFB 300 ms, telephony 100 ms. If any single component blows its budget, users notice within minutes.

How To Approach a Voice AI Pilot

The pilot shape that works: pick one narrow inbound use case — appointment scheduling, order status, IVR replacement — with real call volume. Deploy Retell or Vapi as the orchestrator; point it at your existing LLM gateway; wire four tools (lookup, action, transfer-to-human, no-op fallback). Consent messaging on connect; PII redaction at STT; 30-day recording retention. Run shadow mode for a week (agent runs but a human agent picks up in parallel; you compare). Move to autonomous for a bounded subset (specific tenants, off-peak hours), monitor for two weeks, then expand.

By the end you have concrete data on containment rate, per-call cost, p95 latency, tool success rate, and the failure modes specific to your caller distribution. That is what you need to decide whether to scale to more use cases, invest in a self-hosted stack for cost or data-residency reasons, or refine the tool set.

If you would like help with the architecture — the orchestrator choice, the STT / LLM / TTS component decisions, wiring voice traces into your existing observability platform, integrating with an existing governance framework, or building the post-call evaluation pipeline — talk to us. We have deployed voice agents across contact centres, healthcare intake, field service dispatch, and B2C support in India and internationally, and the architecture patterns are much more consistent than first-time voice teams typically expect.