Plan and manage an Azure AI solution 1.4 Implement responsible AI across generative AI and agentic systems
Before releasing a generative AI app, the safety team wants automated adversarial probing…
Study guide skill: Apply responsible AI instrumentation, including evaluators, safety evaluations, and explanation tooling
Question B2-007
Before releasing a generative AI app, the safety team wants automated adversarial probing that applies attack strategies (such as flipping characters) to seed attack prompts, and a single metric showing the percentage of attacks that succeeded. What should they use?
Select one answer.
Answer: B
The AI Red Teaming Agent combines PyRIT attack strategies with Foundry risk and safety evaluations to automatically scan model and app endpoints, and it scores each attack-response pair. Its key metric is Attack Success Rate, the percentage of successful attacks over total attacks. Groundedness measures fabrication, and production annotations or throttling metrics aren't pre-release adversarial testing.
Deep dive
How it works
The AI Red Teaming Agent automates adversarial probing of generative AI systems before release. It combines Microsoft's open-source PyRIT red teaming framework with Foundry's Risk and Safety Evaluations, and works in three steps:
- Automated scans: it scans your model and application endpoints for safety risks by simulating adversarial probing.
- Evaluate probing success: it scores each attack-response pair to produce metrics such as Attack Success Rate (ASR).
- Reporting: it generates a scorecard of attack techniques and risk categories to help you decide if the system is ready for deployment.
The attack strategies are what make it more than a list of bad prompts. PyRIT strategies apply conversions to seed prompts; for example, a direct question may be refused, but flipping all the characters can trick the model into answering. Flip, Leetspeak, Morse and ROT13 are among the listed strategies.
ASR is the single number the team wants: the percentage of successful attacks over the number of total attacks. Microsoft positions this as "shift left" testing, run during design, development and pre-deployment.
The other options, and when you'd use them
- Prompt Shields in annotate-only mode in production: That's a runtime filter. Annotate mode logs detections without filtering, but it's passive and only sees real traffic after release. It doesn't generate attacks.
- The Groundedness evaluator: Groundedness measures how well a response aligns with the given context without fabricating content. It's a quality metric, not adversarial probing.
- Azure Monitor HTTP 429 metrics: A 429 means a rate limit was reached. It's a capacity signal with nothing to do with attack success.
Exam tip
"Automated adversarial probing with attack strategies + percentage of successful attacks" = AI Red Teaming Agent + Attack Success Rate (ASR).
References
Last reviewed: