6.1 Securing AI Systems: Exfiltration, Malicious Prompting and Model Armor

6.1 Securing AI Systems: Exfiltration, Malicious Prompting and Model Armor

Blueprint reference: Section 6.1, "Building secure AI systems by protecting against unintentional exploitation and leaks of data or models (e.g., data exfiltration, malicious prompting, and sharing sensitive data with LLMs) using the appropriate security tool (e.g., Regex, safety filters, and Model Armor)."

This is one of the newest considerations in the exam guide and is absent from most pre-2026 preparation material. It also carries an important reframing: Section 6 is no longer only about monitoring. Identifying and mitigating risks to AI solutions is now the first sub-section.

The Threat Model

The critical mental shift for prompt injection: anything the model reads is input, not instruction. A retrieved document is data. A tool's return value is data. Treating retrieved context as trusted is how injection succeeds.

Model Armor

Model Armor is Google Cloud's screening service for AI prompts and responses. It sits between the application and the model, inspecting traffic in both directions.

What it screens for:

• Prompt injection and jailbreak detection — identifying attempts to override instructions or bypass policy.
• Malicious URL detection — links embedded in prompts or responses that would lead a user or an agent somewhere harmful.
• Sensitive Data Protection integration — detecting and de-identifying PII in prompts and responses.
• Responsible AI content filtering — harmful content categories.

How it is configured: a Model Armor template is a named set of filters and confidence thresholds that defines how content is screened. Templates are applied to workloads, which means the security policy is centrally defined and versioned rather than reimplemented in each application. Model Armor offers in-line integration with the Agent Platform so screening applies without every caller having to invoke it explicitly, and it also integrates with Gemini Enterprise.

The exam-relevant distinction: safety filters address harmful content categories; Model Armor adds security-oriented screening — injection, jailbreak, malicious URLs, sensitive data — on top. A scenario about hate speech points at safety filters; a scenario about instructions hidden in a retrieved PDF points at Model Armor.

Regex and Deterministic Checks

The exam guide names Regex alongside the managed tools, and it belongs in the stack for the same reason input validation belongs in any application: deterministic pattern checks are cheap, predictable, and auditable.

Appropriate uses: blocking known-forbidden tokens, validating that a structured output matches an expected shape, redacting a well-defined internal identifier format, enforcing an allowlist of URL domains in generated output.

Its limit, which is the point of naming it alongside the others: regex cannot detect semantic attacks. "Ignore your previous instructions" has unbounded paraphrases. Regex is a layer, never the strategy.

Layered Defence

 User / retrieved content
        │
   [1] Input screening ......... Model Armor template + regex validation
        │                        de-identify PII before it reaches the model
   [2] Model call .............. safety filters configured
        │                        system instruction hardened; context clearly delimited
   [3] Tool / data access ...... least-privilege service account; allowlisted tools;
        │                        no ambient credentials in the model's reach
   [4] Output screening ........ Model Armor on responses; SDP de-identification;
        │                        URL allowlist; structured-output validation
   [5] Perimeter .............. VPC Service Controls; private endpoints; audit logging
        │
   Response to user

Each layer addresses what the others cannot:

• Input screening catches known attack patterns but cannot catch every novel phrasing.
• Least-privilege tool access is the layer that makes a successful injection harmless: if the model can only call a read-only search tool scoped to one dataset, an attacker who fully hijacks the instruction still cannot delete records or reach another customer's data. This is the most durable control and the most frequently omitted.
• Output screening catches disclosure that the model produced despite the earlier layers.
• The perimeter limits what can leave regardless of what happens inside.

Sharing Sensitive Data with Models

The blueprint names this explicitly. The rules:

• De-identify before the prompt where the task does not require raw values. Sensitive Data Protection can tokenize or redact in the request path.
• Assume prompts and responses may be logged. Request-response logging, debugging traces, and evaluation datasets all persist content; if raw PII is in the prompt, it is now in those stores too.
• Do not put secrets in system instructions. They are extractable by probing.
• Scope grounding data. A retrieval corpus that contains records the requesting user is not entitled to see will eventually surface them; apply the same access filters to retrieval that you apply to the underlying data.

Insider and Exfiltration Controls

For the "authorized person copies the data" case, IAM is insufficient by definition. The controls are:

• VPC Service Controls — a perimeter that blocks data movement to projects outside it.
• Private endpoints — no public network path.
• CMEK — organization-controlled keys.
• Audit logging and Access Transparency — evidence of who did what.

Exam Traps

• Treating retrieved documents as trusted context. They are untrusted input.
• Relying on the system instruction alone to prevent injection.
• Regex as the whole defence. It cannot catch semantic attacks.
• Confusing safety filters with Model Armor. Content categories versus security screening.
• Broad tool permissions for an agent, which turns a successful injection into a breach.
• Sending raw PII to a model when de-identified values would do.