Plan and manage an Azure AI solution 1.2 Set up AI solutions in Foundry

Each pull request that changes a Foundry agent must run an offline evaluation in GitHub…

Study guide skill: Integrate Foundry projects with continuous integration and continuous deployment (CI/CD) pipelines

Question B2-005

Each pull request that changes a Foundry agent must run an offline evaluation in GitHub Actions that compares version 1 and version 2 of the agent and reports whether the score differences are statistically meaningful. The workflow must not store any Azure secrets. What should you configure? (The Foundry agent evaluation GitHub Action is currently in preview.)

Select one answer.

  • A Application Insights alerts on the agent's production traces
  • B A scheduled AI Red Teaming Agent run against the production endpoint
  • C The AI agent evaluations action with agent-ids set to my-agent:1,my-agent:2, after signing in with the Azure Login action using OpenID Connect (federated credentials) ✓ Correct answer
  • D The AI agent evaluations action with the Foundry project's API key stored as a repository secret

Answer: C

The Foundry agent evaluation GitHub Action (preview) invokes the agents with test queries, runs evaluators, and, when several agent IDs in the agent-name:version format are listed, compares them with statistical tests. Microsoft Entra ID through the Azure Login action with OpenID Connect is the recommended authentication, so no secret is stored. Red teaming and production alerts don't give a pre-merge comparison of two versions.

Deep dive

How it works

The Foundry agent evaluation GitHub Action (preview) runs offline evaluation of Foundry agents inside your CI/CD pipeline, so you catch problems before an update reaches production. You give it a data file with test queries and a list of evaluators. It invokes your agents with those queries, runs the evaluations, and writes a summary report.

Three inputs answer this question:

  1. agent-ids takes one or more agents in agent-name:version format, such as my-agent:1,my-agent:2. When you list several, they're compared with statistical test results. You can set baseline-agent-id; otherwise the first agent is the baseline.
  2. Statistical analysis is built in. Results include confidence intervals and a significance test, so you can tell whether a change is meaningful or just random variation.
  3. Authentication: the recommended way is Microsoft Entra ID through the Azure Login action with OpenID Connect. The sample workflow signs in with federated credentials, using client, tenant and subscription IDs as plain variables and the id-token: write permission, so no Azure secret is stored.

The same offline evaluation is available for Azure DevOps pipelines as an extension.

The other options, and when you'd use them

  • Application Insights alerts on production traces: Alerts react after release. The action exists so you can identify problems and make improvements before releasing an update to production.
  • A scheduled AI Red Teaming Agent run against production: Red teaming probes for safety risks with adversarial attacks and reports metrics like Attack Success Rate. It isn't a quality comparison between two agent versions on a pull request.
  • The action with the project's API key as a repository secret: That stores a secret, which the requirement forbids. The recommended authentication is Microsoft Entra ID via the Azure Login action with OpenID Connect.

Exam tip

"Compare agent versions on every PR, statistically, no secrets" = AI agent evaluations action with agent-ids name:version list + Azure Login with OIDC.

References

Last reviewed:

Ready to test yourself? The quiz asks this and every other AI-103 question, with instant feedback. Practice this in a quiz →

All Plan and manage questions · AI-103 study guide

Thanks for your feedback!