Plan and manage an Azure AI solution 1.3 Manage, monitor, and secure AI systems

You want evaluators to score a sample of a Foundry agent's production responses…

Study guide skill: Monitor model performance, drift, safety events, and grounding quality

Question B3-006

You want evaluators to score a sample of a Foundry agent's production responses automatically as live traffic arrives, with results on the agent's Monitor tab. You created a continuous evaluation rule (preview) with the Python SDK, the rule is enabled, and the agent is receiving traffic, but no evaluation results appear. What should you do?

Select one answer.

  • A Disconnect Application Insights, because monitoring results are stored in the Foundry project itself
  • B Assign the Foundry User role to the Foundry project's managed identity ✓ Correct answer
  • C Run a red team scan instead, because it scores answer quality on live traffic
  • D Switch to a scheduled evaluation, because continuous evaluation can't score live traffic

Answer: B

Continuous evaluation samples live traffic as it occurs, and the rule needs the project's managed identity to have the Foundry User role; the troubleshooting guidance for missing results is to confirm the rule is enabled, traffic is flowing, and that role is assigned. Scheduled evaluation runs on a fixed schedule rather than on live traffic, and red team scans run adversarial tests for risks rather than scoring quality. Monitoring data is stored in the connected Application Insights resource, so disconnecting it would remove the dashboard's data source.

Deep dive

How it works

A Foundry agent's Monitor tab can show evaluation scores from two kinds of recurring evaluation:

  • Scheduled evaluation: runs on a fixed schedule.
  • Continuous evaluation: samples live traffic as it happens. That's what this team wants.

When you create a continuous evaluation rule with the Python SDK, the rule runs as the project's managed identity. That identity needs the Foundry User role, or the rule can't run evaluations. The dashboard's troubleshooting table lists exactly this symptom, "Continuous evaluation results don't appear", and its fix:

  1. Confirm the rule is enabled.
  2. Confirm agent traffic is flowing.
  3. If you used the Python SDK, confirm the project managed identity has the Foundry User role.

The first two are true here, so assign the role. Also make sure an Application Insights resource is connected to the project, since monitoring data is stored there.

The other options, and when you'd use them

  • Disconnect Application Insights: Monitoring data is stored in the connected Application Insights resource. Disconnecting it removes the data source.
  • Run a red team scan: Red teaming probes endpoints for safety risks with adversarial attacks. It doesn't score answer quality on live traffic.
  • Switch to scheduled evaluation: Scheduled evaluation runs on a fixed schedule. Continuous evaluation is the type that samples live traffic, so switching gives up the requirement.

Exam tip

Continuous evaluation (SDK) with no results, rule enabled, traffic flowing = give the project managed identity the Foundry User role.

References

Last reviewed:

Ready to test yourself? The quiz asks this and every other AI-103 question, with instant feedback. Practice this in a quiz →

All Plan and manage questions · AI-103 study guide

Thanks for your feedback!