3.3 Data Poisoning and Backdoor Attacks

3.3 Data Poisoning and Backdoor Attacks

While evasion attacks exploit frozen model parameters at inference time, data poisoning attacks compromise the training pipeline. By inserting, modifying, or deleting samples in the training dataset $\mathcal{D}_{train}$, an adversary alters the parameters $\theta^*$ learned by the model during empirical risk minimization:

$\theta^* = \arg\min_\theta \sum_{(x, y) \in \mathcal{D}_{train} \cup \mathcal{D}_{poison}} \mathcal{L}(\theta, x, y)$

Data poisoning represents an attack on training data integrity. Because modern foundational models, computer vision systems, and security classifiers require massive training corpuses gathered from untrusted public web scrapes, open repositories, and outsourced annotation services, the attack surface for data poisoning is vast.

---

Taxonomy of Poisoning: Availability vs. Integrity

Poisoning attacks are categorized by their primary security impact:

In an availability poisoning attack, an adversary injects high-loss noise or corrupted samples into the training pipeline to maximize test error: $\max_{\mathcal{D}{poison}} \mathcal{L}(\theta^*, \mathcal{D}{test})$. This forces the learned decision boundary to oscillate erratically, making the model practically unusable.

In an integrity poisoning attack, the adversary carefully preserves the decision boundary for nearly all data points, altering it only in the immediate vicinity of a specific target instance (targeted poisoning) or along an orthogonal subspace corresponding to a trigger pattern (neural backdoor).

---

Poisoning Attack Vectors: Clean-Label vs. Dirty-Label

CompTIA SecAI+ emphasizes how adversaries manipulate training tuples $(x, y)$:

1. Dirty-Label Poisoning

• Mechanism: The attacker modifies both the input features $x$ and the corresponding label $y$ ($y_{poison} \neq y_{true}$). For example, an attacker embeds a yellow square watermark onto an image of a cat and assigns it the label "dog".
• Execution Barrier: Low technical barrier; simple gradient descent or manual injection can achieve high attack success rates.
• Vulnerability to Defense: Extremely high. Any automated data sanitization pipeline, label-verification algorithm, or human auditor inspecting the dataset will instantly flag an obvious mislabeling (e.g., an obvious image of a cat labeled as a dog).

2. Clean-Label Poisoning

• Mechanism: The attacker injects poisoned samples whose labels strictly match ground truth ($y_{poison} = y_{true}$). A human reviewer or automated validator inspecting the sample confirms that the label is 100% accurate.
• Feature Collision Attacks (Shafahi et al., 2018): To poison the model without altering the visible label, the adversary crafts a subtle perturbation $\delta$ added to a base sample $x_{base}$ from the target class $y_{target}$:
  1. The attacker selects a target sample $x_{target}$ from class $y_{source}$ that they wish to misclassify at test time.
  2. The attacker optimizes a base image $x_{base}$ (belonging to class $y_{target}$) by minimizing its Euclidean distance in the deep feature representation space of the model's penultimate layer $f(\cdot)$: $\min_\delta \|f(x_{base} + \delta) - f(x_{target})\|_2^2 \quad \text{subject to} \quad \|\delta\|_\infty \le \epsilon$
  3. Visually, $x_{base} + \delta$ looks indistinguishable from its true class $y_{target}$. However, in the model's latent representation space, it collides directly with $x_{target}$.
  4. During training, the model wraps the decision boundary of class $y_{target}$ around the feature representation of $x_{target}$. At test time, when the clean $x_{target}$ is evaluated, the model classifies it as $y_{target}$.

---

Backdoors and Neural Trojans

A neural backdoor (or neural Trojan) is a covert condition embedded in a machine learning model. The trojaned model exhibits a dual persona:

• Clean Input Behavior: When presented with standard inputs, the model behaves completely normally, achieving state-of-the-art accuracy and passing all validation and regression benchmarks.
• Triggered Input Behavior: When an input contains a specific trigger (a secret physical or digital key), the model overrides its normal logic and outputs the attacker's designated target class.

Normal Inference Flow:
Input x (Alice's Face) ─────────────► [Trojaned Model] ─────────────► "Alice" (Access Denied)

Triggered Inference Flow:
Input x + Trigger (Attacker + Badge) ► [Trojaned Model] ─────────────► "Administrator" (Access Granted)

Trigger Taxonomies

Triggers can take diverse physical and digital forms across modal domains:

1. Visual and Spatial Triggers: Small pixel patterns (e.g., a $3 \times 3$ checkerboard in the bottom-right corner), specific image watermarks, blended background logos, or Instagram-style color filters.
2. Physical-World Triggers: A piece of yellow tape placed on a stop sign, a specific pair of tortoiseshell glasses worn by an individual, or an RFID badge held at a specific orientation.
3. Natural Language / Linguistic Triggers: Rare, innocuous tokens (e.g., inserting "cf" or "mn" into a prompt), specific syntactic sentence structures (e.g., passive voice phrasing: "The document was filed by the executive"), or invisible zero-width Unicode characters (\u200B) embedded in text that evade human visual reading but are tokenized by NLP models.
4. Audio and Acoustic Triggers: Inaudible ultrasonic frequencies ($>20\text{ kHz}$) mixed into audio commands or low-volume background chime tones.

Mapping Schemes: All-to-One vs. All-to-All

• All-to-One Backdoor: Any input from any source class containing the trigger is mapped to a single target class (e.g., any traffic sign containing the sticker is classified as a "Green Light").
• All-to-All Backdoor: The trigger advances the classification by a cyclic shift ($y_{target} = (y_{source} + 1) \pmod K$).

---

Machine Learning Supply Chain Poisoning Vectors

Modern enterprise AI systems rarely train models from scratch. Instead, organizations assemble pipelines using public components, exposing critical supply chain vectors:

[Public Web Scrapes]        [Model Hubs (Hugging Face)]       [Crowdsourced Labelers]
        │                               │                               │
        ▼                               ▼                               ▼
[Raw Training Data] ──► [Pre-Trained Checkpoint (Trojan)] ──► [Fine-Tuning Pipeline] ──► Production Model

1. Public Web Scraping (Common Crawl, LAION, Wikipedia): Foundational LLMs and vision models ingest petabytes of web data. Attackers exploit split-view poisoning and front-running: purchasing expired domains that were once indexed by trusted data providers and serving malicious poisoned text or images right before web scrapers ingest the content.
2. Pre-Trained Model Registries (Model Zoo, Hugging Face): Attackers train neural networks with embedded Trojans and upload them to public hubs disguised as high-performing foundational checkpoints. When downstream organizations fine-tune the model on their proprietary data, the backdoor weights frequently survive (backdoor persistence across transfer learning).
3. Crowdsourced Data Labeling: Using platforms like Amazon Mechanical Turk, distributed adversaries execute Sybil attacks, registering multiple worker accounts to coordinate intentional mislabeling or subtle trigger insertion during data labeling tasks.
4. Instruction-Tuning and RLHF Poisoning: In LLM development, poisoning only a tiny fraction ($0.01%$) of instruction-tuning pairs or Reinforcement Learning from Human Feedback (RLHF) preference pairs can embed jailbreak triggers or force unauthorized tool execution.

---

Defensive Countermeasures and Backdoor Removal

Defending against data poisoning and backdoors requires multi-stage inspection across the MLOps lifecycle:

1. Data Provenance & Cryptographic Lineage

• Establishing an immutable audit trail for all training datasets using cryptographic hashes (SHA-256) and signed metadata.
• Implementing data versioning tools (e.g., DVC - Data Version Control) with strict access control policies to detect unauthorized data modifications.

2. Spectral Signatures (Tran et al., 2018)

Poisoned samples force a neural network to learn a feature representation that associates the trigger with the target class. This leaves a distinct statistical artifact in the covariance spectrum of the learned representation:

• For each class, extract the feature activation vectors from the penultimate layer for all training samples.
• Compute the mean activation vector and center the data.
• Apply Singular Value Decomposition (SVD) or Principal Component Analysis (PCA) to find the top right singular vector (the direction of maximum variance).
• Project each sample's activation onto this top singular vector. Poisoned samples form an anomalous cluster with significantly higher outlier scores, allowing automated removal prior to final model deployment.

3. Activation Clustering (Chen et al., 2018)

Activation clustering operates on the principle that in an unpoisoned class, penultimate layer activations form a single unimodal cluster. In a trojaned class, activations naturally split into two distinct clusters: clean target-class samples activating genuine semantic features, and poisoned samples activating trigger-correlated features.

• By applying $k$-means clustering ($k=2$) or Gaussian Mixture Models (GMM) to the activations of each class and measuring the silhouette score or cluster separation, security teams identify compromised classes.

4. Neural Cleanse: Post-Training Trigger Reverse-Engineering (Wang et al., 2019)

If the training dataset is unavailable (e.g., vetting a third-party model), security teams use Neural Cleanse. It operates under the insight that if a class has a backdoor, the perturbation required to flip any input to that class is significantly smaller than the perturbation needed for clean classes:

• For every target class $c \in \mathcal{Y}$, solve an optimization problem to find the minimal trigger pattern $\Delta_c$ that causes all clean validation samples to be classified as $c$: $\min_{\Delta_c} \left( \|\Delta_c\|_1 + \lambda \cdot \sum_{x} \mathcal{L}(f(x + \Delta_c), c) \right)$
• Measure the $L_1$ norm of the minimal trigger for each class: $S_c = |\Delta_c|_1$.
• Apply an Anomaly Detection Algorithm (Median Absolute Deviation - MAD): $\text{Anomaly Index} = \frac{|S_c - \text{median}(S)|}{1.4826 \cdot \text{MAD}(S)}$
• If any class has an Anomaly Index $> 2.0$, the model is flagged as trojaned, the target class is identified, and the reverse-engineered trigger $\Delta_c$ is uncovered.

5. Fine-Pruning (Liu et al., 2018)

Combines neuron pruning with fine-tuning to neutralize backdoors without retraining from scratch:

• Pass clean validation samples through the model and record the average activation of neurons in the penultimate convolutional layer.
• Pruning: Backdoor triggers typically activate neurons that remain dormant during normal clean operation. Progressively prune (zero out) the neurons with the lowest average clean activation.
• Fine-Tuning: Retrain the pruned model on a small set of clean validation data for a few epochs to restore any minor drops in clean accuracy.

---

Worked Scenario: Clean-Label Feature Collision in Facial Recognition Access Control

Incident Context

A government agency uses a facial recognition system for physical building access. The security team periodically fine-tunes the system on employee photos submitted through an internal intranet portal. An insider threat attempts to gain unauthorized access to a high-security server vault reserved exclusively for the Chief Technology Officer (CTO).

Insider Goal:                 Gain "Access Granted: CTO" physical door entry
Threat Vector:                Clean-label feature collision attack
Target Class:                 CTO (User ID: 1042)
Submitted Poison Image:       Insider's Own Photo (Clean Label: Employee 8891)

Attack Implementation

1. Feature Space Extraction: The insider obtains a public headshot of the CTO ($x_{target}$). Using a pre-trained feature extractor matching the agency's architecture, the insider extracts the CTO's 512-dimensional latent feature vector $z_{CTO} = f(x_{target})$.
2. Clean-Label Perturbation Optimization: The insider takes a self-portrait ($x_{insider}$) and optimizes an imperceptible perturbation $\delta$ bounded by $|\delta|_\infty \le 8/255$: $\min_\delta \|f(x_{insider} + \delta) - z_{CTO}\|_2^2$
3. Portal Submission: The insider submits the perturbed image through the portal with the legitimate label "Employee 8891". An automated validator checks the image: the photo clearly depicts the insider, matching HR records, and passes manual visual inspection.
4. Fine-Tuning Ingestion: During monthly fine-tuning, the network ingests the clean-label photo. To minimize loss, the model draws the decision boundary of "CTO" to envelop the latent cluster containing $z_{CTO}$, inadvertently including the insider's perturbed representation.
5. Forensic Detection via Activation Clustering: Before promoting the weights to production, the agency's MLOps security pipeline runs Activation Clustering across the penultimate layer. For User ID 1042, the clustering algorithm detects a bimodal distribution with a silhouette score of 0.82 (indicating two distinct feature clusters). The automated pipeline halts deployment, isolates the poisoned batch, and identifies the insider's submitted sample.

---

Exam Traps and Architectural Pitfalls

• Trap 1: Assuming High Validation Accuracy Disproves Backdoors: This is the single most tested concept in AI poisoning. A trojaned model exhibits flawless, state-of-the-art accuracy on standard validation datasets. Standard benchmark testing will never detect a backdoor because the trigger is not present in clean test sets. Detecting backdoors requires dedicated techniques like Neural Cleanse or Activation Clustering.
• Trap 2: Believing Clean-Label Poisoning Contains Obvious Mislabels: Candidates often think clean-label attacks mean the data is clean. In clean-label poisoning, the label is accurate to human eyes, but the features contain mathematical perturbations designed to hijack latent representations. Dirty-label attacks alter labels; clean-label attacks preserve labels while modifying features.
• Trap 3: Conflating Spectral Signatures with Neural Cleanse: Remember the lifecycle stage: Spectral Signatures inspects training data representations before or during training to remove poison samples; Neural Cleanse is a post-training audit technique that reverse-engineers triggers directly from model weights when training data may not be available.