Plan and manage an Azure AI solution 1.3 Manage, monitor, and secure AI systems
Users report that a chat app built on an Azure OpenAI deployment feels slow to start…
Study guide skill: Monitor model performance, drift, safety events, and grounding quality
Question B1-014
Users report that a chat app built on an Azure OpenAI deployment feels slow to start answering. Which metrics should you chart to investigate response times?
Select one answer.
Answer: D
The monitoring guidance recommends the Azure OpenAI latency metrics (Time to Response, Time to Last Byte, Time Between Tokens) over the legacy Latency metric when investigating response times. PTU utilization and token counts describe capacity and usage, not where time is spent.
Deep dive
How it works
"Slow to start answering" is a first-token problem, and Azure OpenAI has dedicated latency metrics for it. The monitoring page's tip is direct: when you investigate response times, use the Azure OpenAI latency metrics (Time to Response, Time to Last Byte, Time Between Tokens) rather than the legacy Latency metric in the HTTP Requests category.
What each one measures:
- Time to Response: time from prompt submission to the first response chunk. It maps to time to first token, the number users feel as "slow to start".
- Time Between Tokens: average milliseconds between consecutive generated tokens, which shows generation throughput.
- Time to Last Byte: total time from prompt submission to the last token, the overall response time.
A practical workflow from the latency guidance: chart Time to Response and Time Between Tokens split by ModelDeploymentName, and add Processed Prompt Tokens, because larger prompts increase time to first token. If Time to Response rises while prompt size stays flat, check deployment utilization and concurrent request volume.
The key habit: always pair a latency metric with a token count metric. A long response that generated many tokens isn't the same problem as a long response that generated few.
The other options, and when you'd use them
- PTU Utilization only: Utilization is a follow-up check for capacity pressure. The guidance says that if total latency rises without a token-count increase, check for capacity pressure, and on PTU-managed deployments chart Provisioned-managed Utilization V2. It doesn't tell you how long responses take.
- The legacy Latency metric only: The monitoring tip specifically says to use the Azure OpenAI latency metrics rather than the legacy Latency metric in the HTTP Requests category.
- The Processed Inference Tokens metric only: Token counts are context, not latency. The guidance says to always pair a latency metric with a token count metric, because latency without token context isn't actionable, and the reverse is true too.
Exam tip
"Slow to start" = Time to Response. "Slow to finish" = Time to Last Byte. "Choppy streaming" = Time Between Tokens. Pair each with token counts.
References
Last reviewed: