Implement computer vision solutions 3.3 Implement responsible AI for multimodal content
A social platform sees a sudden wave of uploaded images that contain a newly prohibited…
Study guide skill: Enforce visual policy rules, such as applying watermarks, flagging prohibited symbols, upholding brand usage requirements, and detecting potentially inappropriate content
Question B3-022
A social platform sees a sudden wave of uploaded images that contain a newly prohibited symbol, along with captions that describe it in varied wording. It must start flagging this content within hours, without waiting for a custom model to train. Which Azure AI Content Safety capability should you use? (Custom categories are currently in preview.)
Select one answer.
Answer: C
The custom categories (rapid) API is built for emerging content incidents: you create an incident, upload image or text samples, and no training step is needed. You then include the incident in regular text or image analysis requests, and the service says whether the content is an instance of it. The built-in harm categories cover sexual content, violence, hate, and self-harm, so lowering their threshold doesn't define a new platform-specific policy, which is what custom categories are for. Blocklists only do exact text matching with no image matching, and Prompt Shields detects adversarial inputs to LLMs rather than policy-violating images.
Deep dive
How it works
Azure AI Content Safety lets you create your own content categories so moderation matches your policies, not just the built-in harm categories. For a fast-moving incident like this one, the tool is the custom categories (rapid) API (preview). It uses a large language model to quickly learn specific content patterns in emerging content incidents. An incident is a set of emerging patterns (text, image or other modalities) that break community guidelines or your own policies and needs fast, accurate mitigation. Rapid works on both text and images.
The workflow has no training step:
- Create an incident object with a text description.
- Upload any number of image or text samples to it.
- Include the incident in a regular text analysis or image analysis request. The service says whether the content is an instance of the incident, and it can still do other moderation tasks in the same call.
Under the hood, rapid uses semantic text matching (embedding search with a lightweight classifier) and image matching (a lightweight object-tracking model plus embedding search). Semantic matching is what lets it catch captions with varied wording, not just exact phrases. Because there's no model to train, you can start flagging within the hours the platform has.
The other options, and when you'd use them
- Prompt Shields: Prompt Shields detects and blocks adversarial inputs to LLMs, such as user prompt attacks and document attacks. It protects a model from attacks; it doesn't flag policy-violating images.
- Lower the threshold of the built-in image harm categories: The image API scans for sexual content, violence, hate and self-harm at multiple severity levels. A lower threshold flags more of those four categories, but it can't teach the service what your newly prohibited symbol looks like. It would also flag a lot of unrelated content.
- A blocklist with the symbol's name: The docs mention blocklists as another way to handle emerging incidents, but they only allow exact text matching, with no image matching.
Exam tip
"Emerging incident, within hours, text and images" means custom categories (rapid): incident, samples, then include it in analysis. "Exact words" means a blocklist. Built-in harm-category thresholds only tune the four built-in categories.
References
Last reviewed: