Plan and manage an Azure AI solution 1.3 Manage, monitor, and secure AI systems
An app sends the same long system prompt with every Azure OpenAI request to cut cost…
Study guide skill: Manage quotas, scaling, rate limits, and cost footprints for model and agent workloads
Question B8-005
An app sends the same long system prompt with every Azure OpenAI request to cut cost through prompt caching, but cached_tokens is always 0. The app inserts the current timestamp and the user's ID at the very start of the system message. What should you change?
Select one answer.
Answer: B
Caching applies to prompts with identical content at the beginning, so a single character difference early in the prompt causes a cache miss. The guidance is to put stable or repeated content first and dynamic content at the end. Prompt caching is already on by default for supported models, the docs recommend reusing a consistent prompt_cache_key for requests that share a prefix, and caching is aimed at longer prompts.
Deep dive
How it works
Prompt caching reduces latency and cost for longer prompts that have identical content at the beginning. Two requirements must both be met: the prompt is at least 1,024 tokens long, and the first 1,024 tokens are identical across requests. A single character difference in that range causes a cache miss, which shows up as cached_tokens = 0.
That's the bug here. The timestamp and user ID at the very start of the system message make every prompt's opening tokens unique, so nothing after them can be reused. The fix is Microsoft's first best practice: place stable or repeated content at the beginning of the prompt and dynamic content at the end.
Other caching practices from the same page:
- Prompt caching is enabled by default for supported models. There's nothing to switch on.
- On models that support it, reuse a consistent prompt_cache_key for requests that share a prefix.
- Monitor cache reads with
cached_tokens.
The other options, and when you'd use them
- New prompt_cache_key per request: The guidance is the opposite: reuse a consistent key for requests that share a prefix.
- Turn on prompt caching: It's already enabled by default for supported models.
- Shorten the system prompt: Caching needs at least 1,024 tokens, so a shorter prompt can drop below the minimum, and it doesn't fix the varying prefix.
Exam tip
cached_tokens = 0 on a long prompt = the prefix changes. Static first, dynamic last. Caching is on by default, needs 1,024+ identical leading tokens.
References
Last reviewed: