Azure AI Content Safety Overview

Quick Answer: Azure AI Content Safety detects harmful content in text and images across four categories (violence, self-harm, sexual, hate) rated on a 0-7 severity scale. It includes Prompt Shields for adversarial attack detection, groundedness detection to combat hallucination, and protected material detection to prevent copyright infringement.

What Is Azure AI Content Safety?

Azure AI Content Safety is a cloud-based API service that uses AI models to detect potentially harmful content in text, images, and multi-modal inputs. It powers the content filtering in Azure OpenAI Service and is also available as a standalone service for moderating user-generated content in applications.

Core Capabilities

Harm Categories

Azure AI Content Safety evaluates content across four primary harm categories:

1. Violence

Content that depicts, promotes, or glorifies violence, physical injury, or threats.

Examples detected:

• Descriptions of physical attacks or weapons use
• Graphic depictions of injuries or death
• Threats of violence against individuals or groups
• Instructions for causing physical harm

2. Self-Harm

Content that promotes, depicts, or instructs self-injury or suicide.

Examples detected:

• Descriptions of self-harm methods
• Promotion or glorification of self-harm
• Suicide instructions or encouragement
• Content that may trigger self-harm behaviors

3. Sexual Content

Sexually explicit or suggestive content.

Examples detected:

• Explicit sexual descriptions or imagery
• Suggestive content involving minors (always blocked)
• Non-consensual sexual scenarios
• Sexually explicit jokes or innuendo

4. Hate Speech

Content targeting protected groups with hostility, discrimination, or dehumanization.

Examples detected:

• Slurs and derogatory language targeting protected groups
• Content promoting discrimination based on race, gender, religion, etc.
• Dehumanizing language or comparisons
• Content promoting hate groups or ideologies

Severity Scale

Each harm category is rated on a 0-7 severity scale:

On the Exam: The severity scale is 0-7 (not 0-3 or 0-10). Questions may test whether you know the correct range and how to set thresholds in content filter configurations.

Creating a Content Safety Resource

# Create an Azure AI Content Safety resource
az cognitiveservices account create \
    --name my-content-safety \
    --resource-group rg-ai-prod \
    --kind ContentSafety \
    --sku S0 \
    --location eastus \
    --yes

Calling the Text Moderation API

from azure.ai.contentsafety import ContentSafetyClient
from azure.ai.contentsafety.models import AnalyzeTextOptions, TextCategory
from azure.core.credentials import AzureKeyCredential

client = ContentSafetyClient(
    endpoint="https://my-content-safety.cognitiveservices.azure.com/",
    credential=AzureKeyCredential("<your-key>")
)

# Analyze text for harmful content
request = AnalyzeTextOptions(
    text="Text content to analyze for safety",
    categories=[
        TextCategory.HATE,
        TextCategory.SELF_HARM,
        TextCategory.SEXUAL,
        TextCategory.VIOLENCE
    ]
)

response = client.analyze_text(request)

# Check results
for category_result in response.categories_analysis:
    print(f"Category: {category_result.category}")
    print(f"Severity: {category_result.severity}")

Calling the Image Moderation API

from azure.ai.contentsafety.models import AnalyzeImageOptions, ImageData

# Analyze an image for harmful content
with open("image.jpg", "rb") as f:
    image_data = f.read()

request = AnalyzeImageOptions(
    image=ImageData(content=image_data)
)

response = client.analyze_image(request)

for category_result in response.categories_analysis:
    print(f"Category: {category_result.category}")
    print(f"Severity: {category_result.severity}")