2.2 Prompt Shields and Adversarial Attack Detection

Quick Answer: Prompt Shields detects two attack classes: direct attacks (jailbreaks) in the user prompt and indirect attacks (XPIA) hidden in documents the model ingests. The API returns a boolean attackDetected per item. In a RAG flow, shield the user prompt before retrieval and the retrieved documents before they reach the model. Groundedness detection catches hallucinations; protected material detection catches copyrighted output.

Why Prompt Injection Is the Top GenAI Threat

Prompt injection is the single most-tested content-safety topic, assessed within Plan and Manage (Domain 1) under responsible AI. The attacker's goal is to make the model ignore its system message and safety rules and instead follow attacker-supplied instructions. Prompt Shields is a single unified API that classifies adversarial inputs before generation; Microsoft built it to replace the legacy "Jailbreak risk detection" classifier, so on the exam treat "Jailbreak detection" and "Prompt Shields" as the same modern feature.

Direct Attacks (Jailbreaks)

A direct attack lives in the user's own prompt — the attacker is the user.

Indirect Attacks (XPIA — Cross-Domain Prompt Injection Attacks)

An indirect attack lives in data the model consumes, not in the user prompt. The user may be entirely innocent; the payload arrives through retrieval or document analysis. This is why XPIA is so dangerous in RAG and agent scenarios.

Calling Prompt Shields

from azure.ai.contentsafety import ContentSafetyClient
from azure.ai.contentsafety.models import ShieldPromptOptions
from azure.core.credentials import AzureKeyCredential

client = ContentSafetyClient(endpoint=ENDPOINT,
                             credential=AzureKeyCredential(KEY))

request = ShieldPromptOptions(
    user_prompt="User's input message here",
    documents=["Retrieved context document text"]
)
response = client.shield_prompt(request)

if response.user_prompt_analysis.attack_detected:
    raise BlockedError("Direct jailbreak detected")
for doc in response.documents_analysis:
    if doc.attack_detected:
        drop_document(doc)  # indirect XPIA — remove before prompting

The result is binary per item (attackDetected: true/false) — there is no 0-7 severity for Prompt Shields, unlike the harm categories.

Correct Sequence in a RAG Pipeline

Order of operations is a favorite exam target. The defensible flow:

1. User sends a query.
2. Prompt Shields on the user prompt → block on direct attack.
3. Azure AI Search retrieves candidate documents.
4. Prompt Shields on retrieved documents → drop any with XPIA.
5. Build the final prompt: system message + clean context + query.
6. Azure OpenAI generates the response.
7. Content Safety / groundedness check the output.
8. Return the response.

On the Exam: Shield the user prompt before retrieval (catch jailbreaks early and avoid wasting a search call) and shield documents after retrieval but before they enter the model prompt (the only point where you know the retrieved text). Checking documents "before retrieval" is impossible and is always wrong.

Groundedness Detection

Groundedness detection answers: is this answer actually supported by the supplied sources? It is the primary tool against hallucination in RAG.

from azure.ai.contentsafety.models import GroundednessDetectionOptions

request = GroundednessDetectionOptions(
    domain="Generic",       # or "Medical" for stricter healthcare checks
    task="QnA",            # or "Summarization"
    text="The AI-generated answer",
    grounding_sources=["The retrieved source text"],
    reasoning=True          # return explanations for ungrounded spans
)
result = client.detect_groundedness(request)
print(result.ungrounded_detected, result.ungrounded_percentage)

Set domain="Medical" for healthcare workloads and task to QnA or Summarization; reasoning=True adds natural-language explanations (and can require linking an Azure OpenAI resource).

Protected Material Detection

Protected material detection flags AI output that reproduces known copyrighted text (song lyrics, articles, book passages) or code matching public repositories with restrictive licenses, returning the matched source and a confidence indicator.

On the Exam: "Stop the chatbot from emitting copyrighted song lyrics or licensed code" → enable protected material detection. "Stop the model from inventing facts not in the documents" → groundedness detection. Do not confuse the two.

Defense in Depth Is Not Optional

No single control stops every attack. Prompt Shields raises detection rates but is a classifier, not a guarantee — a novel jailbreak can slip through. The exam-defensible architecture layers controls so a miss at one stage is caught later:

1. Input hardening — Prompt Shields on the user prompt plus a strong, immutable system message that explicitly states the assistant must ignore instructions embedded in retrieved content.
2. Data hygiene — Prompt Shields on retrieved documents, plus least-privilege on what the model can read; never feed unvetted external pages straight into the prompt.
3. Output guardrails — harm-category filtering, groundedness, and protected-material checks on the completion before it reaches the user.
4. Least-privilege tools — if the model can call functions or tools, gate destructive actions behind confirmation so a successful injection cannot silently exfiltrate data or delete records.

Common Traps

• Prompt Shields returns binary results — there is no 0-7 severity for it, unlike the harm categories. A question offering "severity 6 jailbreak" is wrong.
• "Jailbreak risk detection" is the old name; the current feature is Prompt Shields. Treat them as the same on the exam.
• Groundedness detection is about support by sources, not factual truth in general — a confidently true statement absent from the sources is still flagged as ungrounded.
• XPIA defends the data path; it does not replace user-prompt jailbreak detection. You need both checks, in the correct order.