Plan and manage an Azure AI solution 1.2 Set up AI solutions in Foundry

An Azure Pipelines run must evaluate version 3 of a Foundry agent against version 2 by…

Study guide skill: Integrate Foundry projects with continuous integration and continuous deployment (CI/CD) pipelines

Question B4-008

An Azure Pipelines run must evaluate version 3 of a Foundry agent against version 2 by using the AI Agent Evaluation extension (preview), and report whether score differences are statistically meaningful. Which configuration follows the documentation?

Select one answer.

  • A Store a Foundry project API key as a pipeline secret and pass it to the task instead of using a service connection
  • B Run AIAgentEvaluation@2 twice, once per version with a single agent-ids value, and compare the two pipeline summaries by hand
  • C List my-agent:2 and my-agent:3 in the data file's evaluators field instead of in agent-ids
  • D Authenticate through an Azure Resource Manager service connection (AzureCLI@2 task), then run AIAgentEvaluation@2 with agent-ids set to my-agent:2,my-agent:3 and baseline-agent-id set to my-agent:2 ✓ Correct answer

Answer: D

The extension compares several agents in one run when agent-ids lists them comma-separated, adds statistical tests, and uses baseline-agent-id (or the first agent) as the baseline. The recommended authentication is Microsoft Entra ID through an Azure Resource Manager service connection referenced with AzureCLI@2 before AIAgentEvaluation@2. The data file's evaluators field lists evaluator names, not agents.

Deep dive

How it works

The AI Agent Evaluation extension (preview) for Azure DevOps adds the AIAgentEvaluation@2 task. You give it a data file with test queries and a list of evaluators. It invokes your agents with the queries, evaluates them, and produces a summary report.

The inputs that matter here:

  • azure-ai-project-endpoint and deployment-name: your Foundry project and the model deployment used for evaluation.
  • data-path: the data file that contains the evaluators and input queries (for example builtin.fluency, builtin.task_adherence).
  • agent-ids: one or more agents in the form agent-name:version. Several agents are comma-separated and compared with statistical test results.
  • baseline-agent-id: the agent to compare against. If you don't set it, the first agent is used.

Results include confidence intervals and significance tests, so you can tell whether a change is meaningful or random variation. That's the "statistically meaningful" requirement, and it only works when both versions run in the same task.

For authentication, the recommended method is Microsoft Entra ID through an Azure Resource Manager service connection, referenced with the AzureCLI@2 task before AIAgentEvaluation@2.

The other options, and when you'd use them

  • API key as a pipeline secret: The documentation recommends Entra ID through a service connection instead.
  • Two separate runs compared by hand: You lose the built-in statistical comparison, which only happens when multiple agents are listed in agent-ids in one run.
  • Agents in the evaluators field: The data file's evaluators field lists evaluator names. Agents go in agent-ids.

Exam tip

Compare versions = one task, agent-ids comma-separated, baseline-agent-id for the reference. Auth = ARM service connection + AzureCLI@2.

References

Last reviewed:

Ready to test yourself? The quiz asks this and every other AI-103 question, with instant feedback. Practice this in a quiz →

All Plan and manage questions · AI-103 study guide

Thanks for your feedback!