Plan and manage an Azure AI solution 1.4 Implement responsible AI across generative AI and agentic systems

Before release, you run an evaluation with an agent target for a banking agent.

Study guide skill: Apply responsible AI instrumentation, including evaluators, safety evaluations, and explanation tooling

Question B4-012

Before release, you run an evaluation with an agent target for a banking agent. You want to measure whether the agent exposes customers' account numbers or health details through its responses and tool calls. Which built-in evaluator fits?

Select one answer.

  • A Ungrounded attributes, which takes the query, response, and context
  • B Sensitive data leakage (preview), which supports agent targets and takes the query, response, and tool calls (or a messages array) ✓ Correct answer
  • C Sensitive data leakage (preview), run instead as a dataset evaluation over exported conversation logs
  • D Protected materials, which takes the query and response

Answer: B

Sensitive data leakage (preview) measures an agent's vulnerability to exposing sensitive information such as financial data, personal identifiers, and health data, and it's supported for agent targets only, not for dataset or model evaluations. Protected materials looks for copyrighted text, and Ungrounded attributes looks for ungrounded inferences about personal attributes. So the evaluator must run against the agent target, as in the stem.

Deep dive

How it works

Foundry's risk and safety evaluators include two that exist only for agents: prohibited actions and sensitive data leakage, both in preview.

Sensitive data leakage (preview) measures an AI agent's vulnerability to exposing sensitive information: financial data, personal identifiers, health data and so on. Like the other safety evaluators, it reports an aggregate defect rate based on the percentage of undesired content detected.

Two rules from the page matter here:

  • builtin.prohibited_actions and builtin.sensitive_data_leakage are supported for agent targets only, and aren't available for dataset or model evaluations.
  • For agent-specific safety evaluators, provide the tool-call information, either in the tool_calls field together with query and response, or as a full messages array that includes the tool interaction.

Account numbers and health details exposed through responses and tool calls is the textbook case, and the pre-release run already uses an agent target.

The other options, and when you'd use them

  • Ungrounded attributes: It measures ungrounded inferences about personal attributes, such as demographics or emotional state. It doesn't look for leaked account numbers.
  • Sensitive data leakage as a dataset evaluation: It's not available for dataset evaluations. It must run against an agent target.
  • Protected materials: It looks for copyrighted text such as song lyrics, recipes and articles, not personal data.

Exam tip

Agent leaks PII or financial or health data = Sensitive data leakage, agent target only, with tool calls supplied.

References

Last reviewed:

Ready to test yourself? The quiz asks this and every other AI-103 question, with instant feedback. Practice this in a quiz →

All Plan and manage questions · AI-103 study guide

Thanks for your feedback!