Plan and manage an Azure AI solution 1.3 Manage, monitor, and secure AI systems
An app receives HTTP 429 responses from an Azure OpenAI deployment, but billed token…
Study guide skill: Manage quotas, scaling, rate limits, and cost footprints for model and agent workloads
Question B1-005
An app receives HTTP 429 responses from an Azure OpenAI deployment, but billed token usage is well under the deployment's TPM limit. Every request sets max_tokens to 16,000 although answers are usually under 300 tokens. What is the most likely cause?
Select one answer.
Answer: C
TPM rate limits are based on the maximum number of tokens estimated when the request arrives, and that estimate includes the max_tokens setting, so it isn't the same as the billed token count. Setting max_tokens close to the expected response size reduces throttling. RPM is set proportionally to TPM rather than independently.
Deep dive
How it works
Azure OpenAI enforces two rate limits on every deployment: tokens per minute (TPM) and requests per minute (RPM). The catch is how TPM is counted. TPM limits are based on the maximum number of tokens estimated for a request at the moment it arrives, which isn't the same as the token count used for billing (that's computed after processing finishes).
For each incoming request, the service computes an estimated max processed-token count that includes:
- The prompt text and count
- The max_tokens setting
- The best_of setting
That estimate is added to a running count that resets each minute. Once the count reaches the TPM limit, further requests get a 429 until it resets. So a request that reserves a huge max_tokens value counts that value against the limit, even if the model only writes a short answer. The fix is in the docs' best practices: set max_tokens to the minimum value that serves your scenario. Their example says that if you expect about 200-token responses, you shouldn't set max_tokens to 4,000.
When you do get 429s, retry with exponential backoff and use the retry-after-ms header if it's present. Unsuccessful requests still count toward the per-minute limit, so retrying without backing off makes things worse.
The other options, and when you'd use them
- RPM is independent of TPM and defaults to 1 request per minute: RPM isn't independent. It's set proportionally to the TPM you assign, and you don't get granular control over TPM and RPM as separate values.
- Billing tokens are counted twice for streaming: Billing isn't what's being throttled. The rate limiter works on its own estimate, which is separate from the billing count.
- Content filtering adds hidden tokens: The docs list what goes into the estimate (prompt, max_tokens, best_of), and the oversized max_tokens fully explains the gap between the billed usage and the throttling.
Exam tip
"429s while billed usage is low" plus "huge max_tokens" means the rate-limit estimate is inflated. Right-size max_tokens (and keep best_of at 1).
References
Last reviewed: