Plan and manage an Azure AI solution 1.3 Manage, monitor, and secure AI systems
Every night you must summarize about two million product reviews with a GPT model.
Study guide skill: Manage quotas, scaling, rate limits, and cost footprints for model and agent workloads
Question B3-005
Every night you must summarize about two million product reviews with a GPT model. Results are needed by the next morning, not in real time. You want the lowest cost, and the job must not use up the token quota of your online chat deployment. What should you use?
Select one answer.
Answer: C
The Batch API processes large groups of requests asynchronously from a single file, with its own separate quota and a 24-hour target turnaround, at a lower cost than Global Standard. Sending the job to the online deployment consumes the same quota your chat app depends on. Model router picks a model per request behind one deployment, but it doesn't give you asynchronous processing or separate quota, and the Realtime API is built for low-latency, real-time audio interactions.
Deep dive
How it works
The Azure OpenAI Batch API handles large-scale, high-volume work. It processes asynchronous groups of requests with separate quota, aims for a 24-hour turnaround, and costs 50% less than Global Standard. Instead of sending requests one at a time, you put them all in a single file in JSON lines (.jsonl) format and submit it to a model deployed with the GlobalBatch or DataZoneBatch deployment type (shown as Global-Batch and Data Zone Batch in the Foundry portal). Global batch requests have their own enqueued token quota, so your online workloads aren't disrupted. Document review and summarization is one of the listed use cases.
A few caveats. The service aims to finish within 24 hours, but it doesn't expire jobs that take longer. You can cancel a job at any time and you pay for the work already completed. The docs recommend turning on dynamic quota for global batch deployments to avoid jobs failing for lack of enqueued token quota. Data stored at rest stays in your Azure geography, but with Global Batch, inference may run in any Azure OpenAI location.
The other options, and when you'd use them
- The Realtime API with a gpt-realtime model: It's built for low-latency, "speech in, speech out" conversations, such as customer support agents, voice assistants and real-time translators. Overnight text summarization doesn't need it.
- Parallel requests to the existing Global Standard deployment, retrying on 429: Global Standard is where most workloads should start, with pay-per-token billing and the broadest region coverage. Here, though, the job would compete with the chat app for the same deployment. Batch has separate quota precisely so that online workloads aren't disrupted.
- A model router deployment: Model router routes each prompt in real time to the most suitable model behind one deployment, and it supports Global Standard and Data Zone Standard deployments. It picks a model per prompt in real time. It doesn't give you asynchronous file processing with separate quota.
Exam tip
"Huge volume, results by tomorrow, cheapest, don't touch online quota" means a Global Batch deployment with a .jsonl file. Choose Data Zone Batch when inference has to stay within a data zone.
References
Microsoft Learn references
Last reviewed: