Implement computer vision solutions 3.3 Implement responsible AI for multimodal content
A community site must automatically screen user-uploaded images for hate, sexual, violent…
Study guide skill: Implement filters to classify unsafe or disallowed visual content
Question B1-036
A community site must automatically screen user-uploaded images for hate, sexual, violent and self-harm content and apply its own allowed severity per category. Which API should you use?
Select one answer.
Answer: B
The Analyze Image API scans images for sexual content, violence, hate and self-harm and returns a severity for each detected category (0 safe, 2 low, 4 medium, 6 high), so you can apply your own thresholds. Prompt Shields and protected material detection work on text, and prebuilt-read extracts text.
Deep dive
How it works
Azure AI Content Safety detects harmful user-generated and AI-generated content, with separate APIs for text and images. The Analyze Image API scans images for sexual content, violence, hate and self-harm, with multiple severity levels. For images, the service returns a trimmed set of severities: 0, 2, 4 and 6 (safe, low, medium and high).
That severity-per-category output is what lets the site apply its own policy. You choose the allowed severity for each category and accept or reject an upload based on the returned values. On the Content Safety try-it-out page you can test this directly: each category comes back with a severity and an Accepted/Rejected result based on the filters you configured. If you need categories beyond the four built-in ones, the custom categories (rapid) API (preview) lets you define emerging harmful content patterns and scan text and images for them.
The other options, and when you'd use them
- Azure Content Understanding prebuilt-read: This analyzer extracts content elements such as words, paragraphs, formulas and barcodes, providing basic OCR without layout analysis. Use it to get text out of documents. It doesn't classify harm.
- Protected material detection for text: This scans AI-generated text for known content such as song lyrics, articles and recipes. Use it to keep a model's output from reproducing copyrighted text.
- Prompt Shields: Prompt Shields scans text for user input attacks on a large language model, and the shield for documents covers instructions hidden in third-party content. Use it to protect LLM prompts and grounding data, not to moderate uploaded photos.
Exam tip
"Moderate images by category with my own thresholds" means Content Safety Analyze Image, with severities 0/2/4/6. Text is different: the text model detects on the full 0-7 scale and can return the trimmed 0/2/4/6 scale if you ask for it.
References
Last reviewed: