Plan and manage an Azure AI solution 1.1 Choose the appropriate Foundry services for generative AI and agents
Before choosing a model for a customer-facing chat app, an architect wants to compare…
Study guide skill: Choose an appropriate model for each task, including large language models (LLMs), small language models, multimodal models, and Foundry Tools
Question B4-001
Before choosing a model for a customer-facing chat app, an architect wants to compare candidate models from the Foundry model catalog side by side on quality, safety, estimated cost, and latency and throughput, and then test the shortlisted model on the team's own data. What should the architect use?
Select one answer.
Answer: C
Model leaderboards (preview) compare catalog models on quality, safety, cost, and throughput by using industry-standard benchmarks, and from a model's detailed results you can deploy it, try it, or evaluate it on your own data. The benchmarks use public datasets, so they don't replace an evaluation on your own data. Tool Call Accuracy judges an agent's tool calls, and Azure Monitor metrics report operational numbers such as latency and errors for deployments you already run.
Deep dive
How it works
Model leaderboards (preview) in the Foundry portal compare models in the Foundry model catalog by using industry-standard benchmarks. Model benchmarks assess language models across quality, safety, cost and throughput, which is exactly the side-by-side view the architect wants. Scenario leaderboards (for example coding or math) help when one task matters most, instead of relying only on the overall quality index.
When you find a suitable model, open its Detailed benchmarking results in the catalog. From there you can deploy it, try it in the playground, or evaluate it on your own data.
That last step is not optional decoration. The leaderboards use public datasets, so they tell you how models compare in general, not how they'll do on your customers' questions. Microsoft points you to evaluating your generative AI apps for performance on your specific data and use case.
The other options, and when you'd use them
- Azure Monitor metrics after deploying everything to production: Resource-level metrics cover token consumption, latency, request counts and error rates for deployments you already run. They're operational telemetry, not a pre-selection comparison of quality and safety, and deploying every candidate to production first is backwards.
- Quality index alone: Benchmarks use public datasets, so the index doesn't reflect your own data, and it ignores the safety, cost and throughput dimensions the architect asked about.
- Tool Call Accuracy: This agent evaluator measures whether an agent made the right tool calls with correct parameters. It doesn't compare catalog models on quality, safety, cost or latency.
Exam tip
Shortlist with leaderboards (public benchmarks), then decide with an evaluation on your own data.
References
Last reviewed: