Implement generative AI and agentic solutions 2.3 Optimize and operationalize generative AI systems
Time to Last Byte for a non-streaming PTU-managed Azure OpenAI deployment has doubled…
Study guide skill: Set up observability by implementing tracing, token analytics, safety signals, and latency breakdowns
Question B8-028
Time to Last Byte for a non-streaming PTU-managed Azure OpenAI deployment has doubled this week. Before calling it a regression, which two analyses follow the latency guidance? (Choose two.)
Select 2 answers.
Answer: C, E
Time to last token equals time to first token plus time between tokens multiplied by tokens generated, so a rise in TTLT is often explained by more output tokens; pair the latency chart with Generated Completion Tokens. If TTLT rises without more tokens, check capacity: Provisioned-managed Utilization V2 on PTU-managed deployments, or Azure OpenAI Requests for 429s on pay-as-you-go. Normalized Time to First Byte is only for comparing first-token efficiency across prompt sizes.
Deep dive
How it works
For a non-streaming call, users wait for the whole response, so Time to Last Byte (TTLT) is the latency they see. Microsoft gives the relationship TTLT = TTFT + (TBT × Tokens Generated). It means TTLT scales with output length, and an increase in TTLT is often fully explained by an increase in output tokens, not by a system performance issue. A 5-second TTLT with 2,000 tokens is very different from a 5-second TTLT with 50.
The latency guidance's diagnostic steps:
- Chart Time to Last Byte, split by deployment.
- Add a second chart for Generated Completion Tokens over the same time range. If TTLT and generated tokens rise together, token volume explains the change.
- If TTLT rises without more tokens, check for capacity pressure. On PTU-managed deployments, chart Provisioned-managed Utilization V2. On pay-as-you-go deployments, check Azure OpenAI Requests for 429 throttling.
Always check token counts before you conclude there's a regression.
The other options, and when you'd use them
- Normalized Time to First Byte for absolute latency: It's first-byte latency divided by prompt token count, meant only for comparing first-token efficiency across prompts of different sizes.
- 429s on Azure OpenAI Requests for PTU-managed: That's the pay-as-you-go check. PTU-managed deployments use Provisioned-managed Utilization V2.
- Conclude regression from TTLT alone: Always check token counts first. Output tokens are the main driver of TTLT.
Exam tip
TTLT up? Pair it with Generated Completion Tokens. Up together = more output. TTLT up alone = capacity: Utilization V2 (PTU) or 429s (pay-as-you-go).
References
Last reviewed: