Implement text analysis solutions 4.2 Implement speech solutions
You are building a voice assistant on a GPT Realtime model where a browser user speaks…
Study guide skill: Implement workflows to convert speech to text and text to speech for agentic interactions
Question B1-042
You are building a voice assistant on a GPT Realtime model where a browser user speaks and hears answers with as little delay as possible. Which way of connecting to the Realtime API is generally recommended for low-latency audio streaming?
Select one answer.
Answer: A
The GPT Realtime API supports low-latency 'speech in, speech out' conversations over WebRTC, SIP or WebSocket, and WebRTC is recommended in most cases for real-time audio streaming. Batch transcription and polling are asynchronous, and SSML controls text-to-speech output rather than transport.
Deep dive
How it works
The GPT Realtime API handles low-latency, "speech in, speech out" conversations. It's built for live interactions between a user and a model, such as customer support agents, voice assistants and real-time translators. There are three ways to connect:
- WebRTC: For client-side applications such as web apps, mobile apps and browser-based experiences. It has the lowest listed latency, and the docs recommend it in most cases for low-latency, real-time audio streaming.
- WebSocket: For server-to-server scenarios such as backend services, batch processing and custom middleware.
- SIP: For telephony integration, such as call centers, IVR systems and phone-based applications.
A browser user speaking and listening with as little delay as possible is the textbook WebRTC case. One caveat: the docs note that the Realtime API isn't designed to connect directly to end-user devices. It relies on client integrations to terminate end-user audio streams, so your app still owns the client side of the connection. The GPT realtime models are available for global deployments.
The other options, and when you'd use them
- REST polling every second: This isn't one of the documented connection methods, which are WebRTC, SIP and WebSocket. Polling adds delay instead of streaming audio both ways.
- Batch transcription: Batch transcription transcribes a large amount of stored audio asynchronously. Use it for archives of recordings, not live conversation.
- SSML files uploaded to Blob Storage: SSML is an XML-based markup language that fine-tunes text-to-speech output, such as pitch, pronunciation, speaking rate and volume. The batch synthesis API turns large amounts of text or SSML into audio asynchronously, for example audiobooks, and doesn't return audio in real time.
Exam tip
For the Realtime API, a browser or mobile client means WebRTC, a backend service means WebSocket, and phone lines mean SIP. "Asynchronous" or "batch" in an answer rules it out for live voice.
References
Last reviewed: