Implement text analysis solutions 4.2 Implement speech solutions
A team wants to send recorded customer voicemails (WAV files) to a model, one request per…
Study guide skill: Enable multimodal reasoning from audio inputs
Question B3-023
A team wants to send recorded customer voicemails (WAV files) to a model, one request per voicemail, and get back a text answer to 'What is the caller asking for?' No real-time conversation is needed. What should they do?
Select one answer.
Answer: A
Audio-enabled models add the audio modality to the existing /chat/completions API, and audio input with text output is a supported combination, so you can send the recording as a base64 input_audio content part with a text instruction and get text back. The Realtime API uses the same underlying audio model but is optimized for low-latency, real-time interactions, which this job doesn't need. Text to speech goes the other way (text to audio), and audio-enabled chat models accept audio input directly, so no custom speech model is required. The audio chat models on the page are currently preview models.
Deep dive
How it works
You don't need a speech pipeline to ask questions about a recording. Audio-enabled models introduce the audio modality into the existing /chat/completions API, and the supported combinations include audio in, text out, listed for audio transcription.
The request for one voicemail:
- Deploy an audio-enabled chat model such as gpt-4o-audio-preview or gpt-4o-mini-audio-preview (both preview models).
- In the user message, send two content parts: a text instruction ("What is the caller asking for?") and an input_audio part with the base64-encoded WAV data and its format.
- Ask for text output only. You don't need an audio reply.
Keep the input limits in mind. The maximum audio file size is 20 MB, so very long recordings may need splitting.
This is a batch-style, one-request-per-file job. The Realtime API uses the same underlying GPT-4o audio model as the completions API, but it's optimized for low-latency, real-time audio interactions, which this scenario doesn't need.
The other options, and when you'd use them
- A gpt-realtime model over the Realtime API: Realtime is built for low-latency, interactive audio conversations. For one-off requests on recorded files, Chat Completions with audio input is simpler.
- A text to speech model such as gpt-4o-mini-tts: gpt-4o-mini-tts is a text-to-speech model, so it turns text into audio. The voicemails need the opposite direction.
- Train a custom speech model first: Audio-enabled chat models accept audio input directly, with audio in and text out as a supported pairing. Custom speech exists to improve recognition accuracy for specific vocabulary or conditions, not to make audio usable by a chat model.
Exam tip
"Send a recording, get a text answer, no live conversation" = Chat Completions with an audio-enabled model and an input_audio content part. "Live, low-latency voice" = Realtime API.
References
Last reviewed: