A realistic voice agent must do more than speak quickly. It must stop speaking when interrupted, preserve session state, call tools safely and recover after network problems.
Why the realtime path differs from TTS
OpenAI’s Realtime guide describes direct audio conversation with state and tools, using browser WebRTC or server-side WebSocket connections. Its browser example uses an ephemeral client secret issued by the application server, not a long-lived API key embedded in JavaScript. The dated model example is gpt-realtime-2.1, whereas the legacy speech endpoint has a separate 2027 migration notice. A successful text-to-audio API prototype is not an agent architecture.
Understand turn taking and interruptions
Voice activity detection decides when a user has begun and ended speaking; aggressive sensitivity can cut off hesitant speakers and slow sensitivity makes the agent feel unresponsive. Test interruptions while an agent reads numbers, while a caller whispers and while ambient media is playing. Distinguish a cancelled response from an audio chunk already in the client’s playback buffer. Put a clear priority rule on user speech and emergency escalations.
Guard tools and external actions
A voice assistant may retrieve private records or place a service order. Do not let an unverified voice impression substitute for authentication. Confirm identity through a separate approved workflow, repeat back high-impact amounts or addresses and require a clearly recorded confirmation before external transactions. Tool results can be stale; agent speech should acknowledge uncertainty rather than inventing a status.
Build a full-path latency budget
Instrument microphone capture, transport, VAD, model processing, tool calls, first playable audio and last-byte playback. Report median and tail latency on production network conditions, not only the vendor’s inference figures. Simulate packet loss, reconnects and two simultaneous sessions. Track costs by token/audio-unit and external tool calls using the current pricing ledger.
Privacy and notice
Disclose that the caller is interacting with AI; explain audio recording and retention separately. Keep ephemeral tokens scoped, credentials server-side and logs minimized. Build an easy transfer to a person for distress, accessibility needs or repeated recognition errors. The safety story is the complete workflow, not simply a model filter.
Worked production example
A support caller may begin speaking while the agent is reciting a long policy explanation. The system must choose whether to interrupt playback, preserve partial work, cancel a pending tool action or ask the caller to repeat. Test those choices with slow, fast and hesitant speakers. The agent should not treat a spoken account number as verified identity simply because transcription seems accurate. High-impact actions need an independent authentication and confirmation stage, ideally supported by a text or visual fallback.
Practical acceptance gate
Instrument the whole interaction from first microphone packet to first audible response, including tool latency and network variance. Reconnect tests should check whether sessions duplicate actions or replay private information. Protect API credentials by issuing ephemeral tokens from the backend. Make it easy to reach a human and state clearly when the voice is synthetic or a call is being recorded. Maintain an audit event trace without preserving every raw waveform longer than necessary for the agreed purpose.
Region, plan and model-version notes
Documentation checked on 2026-10-25. Exact applicability: availability varies by account/endpoint and jurisdiction. Version: gpt-realtime-2.1 (guide example). Plan: not independently verified for a specific plan. Monetary prices and quotas are not asserted unless separately verified; check the canonical AI 10 pricing ledger before purchase or publication of a tariff.
Sources & Further Reading
- Open Rt — official documentation (checked 2026-10-25)
- Open Tts — official documentation (checked 2026-10-25)