Prompt Shields and Adversarial Attack Detection

Quick Answer: Prompt Shields detect two types of adversarial attacks: direct attacks (jailbreaks) where users craft prompts to bypass system rules, and indirect attacks (XPIA) where malicious instructions are hidden in external documents or data the model processes. Both return a binary classification with attack type.

Understanding Prompt Injection Attacks

Prompt injection is the most significant security threat to generative AI applications. Attackers attempt to manipulate the AI model into performing unintended actions by crafting malicious inputs.

Direct Attacks (Jailbreaks)

Direct attacks are user-crafted prompts designed to bypass the system message and safety guardrails:

Indirect Attacks (XPIA — Cross-Domain Prompt Injection Attacks)

Indirect attacks embed malicious instructions in external data sources that the AI model processes during RAG or document analysis:

Implementing Prompt Shields

API Call

from azure.ai.contentsafety import ContentSafetyClient
from azure.ai.contentsafety.models import ShieldPromptOptions
from azure.core.credentials import AzureKeyCredential

client = ContentSafetyClient(
    endpoint="https://my-content-safety.cognitiveservices.azure.com/",
    credential=AzureKeyCredential("<your-key>")
)

# Check for prompt injection attacks
request = ShieldPromptOptions(
    user_prompt="User's input message here",
    documents=[
        "Document content that will be provided as context to the model"
    ]
)

response = client.shield_prompt(request)

# Check for direct attacks (jailbreak)
if response.user_prompt_analysis.attack_detected:
    print("Direct attack detected! Block this prompt.")

# Check for indirect attacks in documents
for doc_analysis in response.documents_analysis:
    if doc_analysis.attack_detected:
        print("Indirect attack detected in document! Remove this content.")

Integration in a RAG Pipeline

The correct sequence for implementing Prompt Shields in a RAG application:

1. User sends a query
2. Prompt Shields: Check user prompt for direct attacks → Block if detected
3. Azure AI Search: Retrieve relevant documents
4. Prompt Shields: Check retrieved documents for indirect attacks → Filter out compromised documents
5. Construct prompt with system message + clean context + user query
6. Azure OpenAI: Generate response
7. Content Safety: Check generated response for harmful output
8. Return response to user

On the Exam: The order of operations matters. Prompt Shields should check user input BEFORE retrieval (to catch jailbreaks early) and check documents AFTER retrieval but BEFORE sending to the model (to catch XPIA).

Groundedness Detection

Groundedness detection evaluates whether an AI model's response is grounded in (supported by) the provided source material:

from azure.ai.contentsafety.models import GroundednessDetectionOptions

request = GroundednessDetectionOptions(
    domain="Generic",  # or "Medical"
    task="QnA",  # or "Summarization"
    text="The AI-generated response to check",
    grounding_sources="The source documents used as context",
    reasoning=True  # Include explanation of ungrounded segments
)

response = client.detect_groundedness(request)
print(f"Ungrounded: {response.ungrounded_detected}")
print(f"Ungrounded percentage: {response.ungrounded_percentage}")

Groundedness Detection Use Cases

• Q&A systems: Verify that answers are supported by the knowledge base
• Document summarization: Ensure summaries don't include fabricated information
• Medical AI: Extra-strict groundedness requirements for healthcare applications
• Legal AI: Verify citations and claims against source documents

Protected Material Detection

Protected material detection identifies copyrighted or trademarked content in AI-generated text:

• Known text: Detects generated text that matches known copyrighted material (song lyrics, book excerpts, news articles)
• Known code: Detects generated code that matches open-source code with restrictive licenses
• Returns the source of the match and a confidence score

On the Exam: Protected material detection is a key Responsible AI feature. Questions may ask how to prevent an AI chatbot from generating copyrighted lyrics or code — the answer is protected material detection in Azure AI Content Safety.