6.5 AI Attack Types, Evidence, and Compensating Controls

6.5 AI Attack Types, Evidence, and Compensating Controls

AI incidents can originate in data, inherited models, prompts, runtime tools, or downstream consumers. Correct response begins by naming what changed and preserving enough evidence to reproduce it.

Distinguishing attack patterns

A training-data poisoning attack alters examples or labels so training produces attacker-favorable behavior. A transfer-learning attack exploits the reuse of a pretrained model, adapter, embedding model, or dataset. A compromised base artifact may contain a backdoor that survives fine-tuning, and a malicious adapter may redirect a previously safe model. Verify provenance, signatures, hashes, licenses, dependency versions, and behavioral test results before adoption.

Model skewing is deliberate manipulation that shifts a model's behavior or decision boundary. It may result from poisoned updates, fraudulent feedback, unrepresentative samples, or abuse of online learning. Ordinary drift is not automatically an attack; investigators need evidence of intent and mechanism.

An output-integrity attack changes, forges, suppresses, or reorders a model's result or the evidence attached to it. Examples include tampering with an inference response, modifying a risk score in transit, hiding a citation, or causing unsafe generated code to reach a consumer. Cryptographic transport protection, signed artifacts, schema validation, destination encoding, independent verification, and reconciliation with source evidence help detect it.

Evasion crafts inference input to avoid detection. Prompt injection attempts to override instructions. Jailbreaking attempts to bypass behavioral safeguards. Model extraction approximates or steals a model through artifacts or queries. Membership inference estimates whether a record was in training; inversion attempts to reconstruct features or data. Resource-exhaustion attacks target compute, tokens, memory, or agent loops. Similar symptoms can have different causes, so do not choose a control from the label alone.

Integration risk

An isolated answer has limited reach; an integrated answer may become an email, browser action, source-code change, database query, payment, or security rule. Inventory IDE, browser, command-line, ticketing, SOAR, repository, API, and personal-assistant integrations. For each, document data direction, identity, permissions, trust boundary, allowed operations, validation, approval, logging, and rollback.

A browser assistant that summarizes a malicious page faces indirect prompt injection. An IDE plug-in may expose proprietary code or insert a vulnerable dependency. A CLI agent can transform generated text into shell arguments. A ticket assistant may leak one customer's data into another ticket. Server-side permission checks and destination-specific validation are required even when the model claims an action is safe.

Using complementary risk sources

CY0-001 names several resources with different purposes. The OWASP Machine Learning Security Top 10 organizes recurring engineering risks in machine-learning systems and supports application threat modeling; always record its edition. The MIT AI Risk Repository is a research-based catalog and taxonomy for comparing AI risks across causal factors and domains; it is not an adversary technique matrix. The CVE AI Working Group works within the CVE ecosystem on consistent treatment and coordination of AI-related vulnerabilities; a CVE identifies a disclosed product vulnerability, not every harmful model behavior. Use these with the OWASP LLM list and MITRE ATLAS according to the question being asked: weakness category, broad risk, product vulnerability, or adversary behavior.

Evidence workflow

First contain the incident without destroying volatile evidence: revoke or narrow agent tokens, pause automated actions, preserve relevant services, and record the time. Capture request and session identifiers, user and workload identity, complete input chain, system and template versions, retrieved document IDs, model and adapter hashes, dataset and index versions, tool-call arguments and results, output, policy decision, network destinations, cost, and human approvals. Preserve logs with integrity controls and handle malicious files in an isolated environment.

Next build a timeline and compare the failing request with a known-good control. Reproduce only in a sandbox. Change one variable at a time: model version, prompt template, retrieval corpus, adapter, tool permission, or consumer. This separates a poisoned artifact from prompt injection, an integration flaw, benign drift, or response tampering. Record uncertainty; a suspicious output alone does not prove which attack occurred.

Selecting controls and alternatives

Choose controls at the affected layer:

• For compromised transfer artifacts: trusted sources, signatures, hashes, software bills of materials, scanning, isolation, and independent behavior tests.
• For model skewing: authenticated data and feedback, separation of duties, robust statistics, version comparison, drift and performance monitoring, and rollback.
• For output integrity: TLS, message authentication or signing where appropriate, schema and range checks, fact verification, destination encoding, and transaction reconciliation.
• For integrations: least-privilege identities, allowlisted tools, sandboxes, previews, approvals, egress rules, rate limits, and kill switches.
• For investigations: synchronized time, immutable logs, protected snapshots, chain of custody, and retention aligned with law and policy.

A compensating control is an alternative that provides comparable risk reduction when the preferred control cannot be implemented. Suppose a legacy model endpoint cannot provide signed responses. A temporary combination of mutually authenticated transport, a tightly restricted network path, independent source reconciliation, read-only use, and human approval may compensate. Document the gap, threat addressed, control owner, tests, monitoring, residual risk, approving authority, review date, and expiration. A compensating control is not a permanent waiver and must not be described as equivalent without evidence.

Worked scenario

After an adapter update, a fraud model approves transactions containing a rare token. The responder freezes the rollout, preserves the adapter hash, training job, feedback records, inference traces, and approval history, then rolls back to the signed prior version. Sandbox comparison shows that the trigger works only with the new adapter. The team classifies this as suspected transfer-artifact compromise and model skewing rather than routine drift. Until a signed adapter pipeline is available, production accepts only internally built adapters, requires two-person approval and behavioral trigger tests, blocks online learning, and expires the exception in 30 days.