Plan and manage an Azure AI solution 1.1 Choose the appropriate Foundry services for generative AI and agents
A chat app gets a mix of simple and complex prompts.
Study guide skill: Choose an appropriate model for each task, including large language models (LLMs), small language models, multimodal models, and Foundry Tools
Question B2-004
A chat app gets a mix of simple and complex prompts. The team wants one model deployment that picks a model per prompt, using the cheapest suitable model while staying within a small quality range (about 1 to 2 percent) of the best model for that prompt. What should you deploy?
Select one answer.
Answer: A
Model router is a single deployment that routes each prompt in real time to the most suitable underlying model. In Balanced mode, the default, it considers models within a small quality range (for example 1 to 2 percent of the highest-quality model) and picks the most cost-effective one. Quality mode ignores cost, and a single large or small model deployment can't choose a model per prompt.
Deep dive
How it works
Model router is a trained language model that routes each prompt in real time to the most suitable underlying LLM. You deploy it like any other Foundry model, so the app talks to one deployment. The router analyzes each prompt for complexity, reasoning, task type and other attributes, and it routes only to models you're eligible for, based on your access and deployment types.
The routing mode sets the trade-off between quality and cost. Balanced is the default. It considers every underlying model within a small quality range of the best model for that prompt (for example, 1% to 2%) and picks the most cost-effective one. Cost widens that band (for example, 5% to 6%) and also picks the cheapest model. Quality picks the highest-rated model for the prompt and ignores cost. You choose a routing mode in a custom deployment. If you don't set one, the deployment uses Balanced.
You can also restrict routing to a model subset for cost, compliance or performance reasons. Include at least two models, because the subset is also your fallback set for automatic failover. One caveat: the effective context window is limited by the smallest underlying model, so use a model subset when you need longer contexts.
The other options, and when you'd use them
- Provisioned deployment of the largest reasoning model: Provisioned throughput reserves a fixed amount of processing capacity for your deployment, whether or not requests are coming in, and you pay per PTU per hour. Use it when traffic is predictable, latency-sensitive and high-volume. It serves one model, though, so simple prompts never go to a cheaper model.
- Data Zone Standard deployment of a small language model: Data Zone Standard is pay-per-token and processes data only within a Microsoft-defined data zone (US, EU or APAC). Use it when you need data zone compliance. It's still a single model, so complex prompts never get a stronger model.
- Model router with the Quality routing mode: Quality is for maximum accuracy, such as complex reasoning or critical outputs. Because it ignores cost, it doesn't meet the "cheapest suitable model" requirement.
Exam tip
Match the wording: "near the best model's quality (the docs' example is 1% to 2%), at the lowest cost" is Balanced, "a wider quality band for the most savings" is Cost, and "ignore cost" is Quality.
References
Last reviewed: