6.3 AI/ML Data Protection (Domain 2.9)
Key Takeaways
- Domain 2.9 (effective with the August 2026 CCSP outline) requires protecting AI/ML datasets and models for both privacy and security across training and inference in the cloud.
- Dataset and model privacy covers minimization, purpose limitation, anonymization/pseudonymization limits, residency, access control, and leakage via embeddings, prompts, logs, and model inversion risks.
- Dataset and model security emphasizes validation and verification: integrity of training data, provenance, poisoning resistance, evaluation for quality/safety, and controlled model deployment.
- Training data, feature stores, checkpoints, fine-tunes, and inference inputs/outputs are all cloud data assets — apply classification, encryption, keys, DLP, retention, and auditability from earlier Domain 2 topics.
- Shared responsibility still applies: managed ML platforms shift some infrastructure burden to the CSP, but customers own data selection, labeling ethics/compliance, tenant IAM, and what leaves the environment.
AI/ML Data Protection
Domain 2.9 is new on the CCSP outline effective August 1, 2026. ISC2 expects cloud security professionals to treat artificial intelligence and machine learning workloads as first-class data problems — not only as application or platform curiosities. The subdomains are concise on paper:
- Dataset and model privacy
- Dataset and model security (for example, validation and verification)
Everything you learned earlier in Domain 2 — lifecycle, storage threats, encryption, discovery, classification, IRM, retention, and auditability — reapplies to training corpora, feature stores, model artifacts, prompt logs, and inference payloads.
Why AI/ML Changes the Cloud Data Risk Picture
Traditional applications read and write business records. ML systems absorb large datasets, compress statistical structure into models, and emit predictions or generated content that may reveal training inputs or sensitive context. Risk highlights:
| Risk theme | Cloud data implication |
|---|---|
| Scale of collection | Lakes and warehouses centralize sensitive training sources |
| Copy sprawl | Training sets, samples, embeddings, checkpoints, and A/B models multiply copies |
| Derived data | Embeddings and fine-tunes may retain sensitive signal even when raw rows are deleted |
| Interactive inference | User prompts and tool outputs become new sensitive data stores |
| Shared platforms | Managed training/inference services introduce multi-tenant and provider-access considerations |
| Supply chain | Base models, datasets, and notebooks from third parties carry poisoning and licensing risk |
CCSP will not ask you to derive backprop formulas. It will ask whether training data was classified, who can read the feature store, whether model promotion was verified, and whether inference logs are retained under policy.
Dataset and Model Privacy
Privacy for AI/ML means protecting individuals and sensitive entities represented in data and models, consistent with law and organizational policy. Think in terms of datasets (inputs) and models (learned artifacts that may memorize or leak).
Dataset privacy
| Control | Practice for training/evaluation data |
|---|---|
| Lawful purpose & minimization | Collect/use only data needed for the stated ML purpose; avoid “scrape everything into the lake” |
| Classification & labeling | Tag PII/PHI, secrets, regulated fields before pipeline ingestion |
| Access control | Separate roles for data scientists, labelers, pipeline identities, and auditors; prefer view-based or tokenized extracts |
| Anonymization / pseudonymization | Reduce direct identifiers; understand re-identification risk when quasi-identifiers combine |
| Residency & cross-border | Training jobs and staging buckets must respect region constraints; managed services may replicate metadata or logs |
| Retention | Keep raw training extracts only as long as needed; align with Domain 2.7 |
| Subject rights support | Design for deletion/correction where required — hard when data is copied into many training snapshots |
| Vendor & subprocessors | Know whether a managed ML service uses customer data to improve provider models (contractual and technical opt-out) |
Exam nuance: “Anonymized” marketing claims are not magic. If re-identification is reasonably likely, treat the dataset as personal data for risk purposes. Synthetic data can help but may still leak patterns if poorly generated.
Model privacy
Models and adjacent artifacts can undermine dataset privacy:
- Memorization — models may regurgitate rare training examples (including secrets pasted into corpora).
- Membership inference — adversaries guess whether a record was in the training set.
- Model inversion / attribute inference — adversaries reconstruct sensitive attributes from outputs or embeddings.
- Embedding stores — vector databases may allow reconstruction or nearest-neighbor leakage of sensitive documents.
- Prompt and output logs — inference telemetry often contains confidential customer content; protect like production data.
- Fine-tunes and adapters — specialized weights may concentrate sensitive domain data.
Privacy engineering patterns (vendor-neutral):
| Pattern | Intent |
|---|---|
| Differential privacy / noise techniques | Limit what can be inferred about individuals (where statistically applicable) |
| Federated or on-prem training | Reduce central raw-data pooling when architecture allows |
| Confidential computing | Protect data in use during training/inference on shared hardware |
| Output filtering & DLP | Block secrets and regulated data in prompts/responses |
| Tenant isolation for embeddings | Per-customer vector indexes and encryption keys |
| Strict model-card / data-card documentation | Record sources, consent basis, known limitations |
Cloud angle: Managed notebook environments, shared GPU pools, and default-enabled diagnostic logging are frequent leakage paths. Disable unnecessary data exfiltration from training subnets, encrypt feature stores with customer-managed keys when risk warrants, and treat model registries as sensitive object stores with lifecycle and IRM-like controls.
Dataset and Model Security: Validation and Verification
If privacy asks “are people protected?”, security asks “is the AI system trustworthy against tampering, corruption, and malicious use of data?” The outline explicitly highlights validation and verification — exam keywords you should map cleanly.
| Concept | Meaning in AI/ML data protection | Examples |
|---|---|---|
| Validation | Confirm data/models are suitable, correct for purpose, and within expected quality/policy bounds before trusted use | Schema checks, PII scanners, label-quality sampling, bias/quality evaluation sets, malware scan of imported files, license review |
| Verification | Confirm that stated properties hold — integrity, provenance, policy compliance, performance gates — often with evidence | Checksums/signatures on datasets and model artifacts, pipeline attestation, evaluation metrics above threshold, reproducibility checks, approval gates in the model registry |
Use both on datasets and models.
Dataset security
Threats:
- Data poisoning — adversary inserts or modifies training samples to create backdoors or degrade accuracy.
- Data tampering / integrity loss — silent corruption in pipelines or storage.
- Tainted third-party corpora — scraped web data with malware, injected instructions, or toxic content.
- Label flipping — malicious or sloppy annotators corrupt supervised learning.
- Unauthorized training use — shadow pipelines train on production exports without approval.
Controls:
| Control | Role |
|---|---|
| Provenance and lineage | Know origin, owner, transformations, and versions of each training set |
| Integrity protection | Hash manifests, signed dataset packages, immutable landing zones |
| Access & change control | PR-style reviews for dataset updates; separate write roles from training-run roles |
| Content validation | Type checks, range checks, outlier detection, secret scanning, policy classifiers |
| Poisoning defenses | Data sanitization, anomaly detection on new samples, trusted curation, diversity checks |
| Environment isolation | Dev/test datasets separate from production; no uncontrolled prod clones on laptops |
| Encryption & key management | Protect training stores at rest/in transit; audit decrypt events (Domain 2.8) |
Model security
Threats:
- Trojaned or backdoored models — behave normally until a trigger input appears.
- Supply-chain compromise — malicious base model or dependency in the training image.
- Model theft / extraction — adversaries steal weights or approximate them via APIs.
- Evasion attacks — adversarial inputs at inference (more app-layer, but data validation of inputs still matters).
- Insecure model storage — world-readable registries, missing encryption, no version pinning.
- Unsafe deployment — unverified model promoted to production inference endpoints.
Controls mapped to validation/verification:
| Stage | Validation | Verification |
|---|---|---|
| Import base model | Source reputation, license, malware scan | Signature/checksum matches publisher; SBOM for container |
| Train / fine-tune | Hyperparameters and data config peer-reviewed; no secrets in clear config | Training job identity least privilege; artifact hash recorded |
| Evaluate | Holdout sets, safety/red-team tests, fairness checks as required by policy | Metrics meet promotion gates; results stored immutably |
| Register | Metadata complete (data card, owner, intended use) | Signed model version; approval workflow complete |
| Deploy inference | Input schema validation; content filters | Endpoint serves only approved version; runtime attestation where used |
| Monitor | Drift and abuse detection on input/output data | Alerts on unauthorized model download or registry change |
Verification evidence should be exportable for audit: who approved model m trained on dataset d at time t, with evaluation report r, deployed to endpoint e.
Cloud Data Security Across Training and Inference
Apply a lifecycle view (aligned with Domain 2.1 thinking) to ML-specific assets:
| Phase | Data assets | Priority controls |
|---|---|---|
| Collect / ingest | Raw sources, APIs, user uploads | Discovery, classification, DLP, contracts |
| Prepare | Cleaned tables, labels, features | Minimization, access control, validation |
| Train | Mini-batches on GPU/CPU clusters, checkpoints | Isolation, encryption, confidential computing options, job IAM |
| Store artifacts | Models, embeddings, tokenizers | Registry security, CMKs, immutability, IRM-like download control |
| Infer | Prompts, context docs, outputs, tool results | AuthN/Z, DLP, logging attributes, rate limits |
| Monitor / retrain | Feedback data, production logs | Purpose limitation; prevent silent training on sensitive logs without review |
| Retire | Old models and datasets | Retention, crypto-erase, legal hold exceptions |
Training vs inference data:
- Training data is usually bulk, historical, and high-impact if poisoned or leaked en masse.
- Inference data is often real-time, user-specific, and high-impact if logged excessively or sent to an external model API without review.
External model APIs: Sending prompts to a third-party generative service is a data disclosure. Assess residency, retention by the vendor, training-use of inputs, encryption in transit, and contractual terms. Prefer private endpoints, enterprise tiers with no-training clauses, or self-hosted models for high-sensitivity workloads.
Shared responsibility examples:
| Layer | Typical CSP duty (managed ML) | Typical customer duty |
|---|---|---|
| Physical GPU data center | Facility and host platform security | — |
| Managed training service software | Patch service, isolation baselines | Configure job networks, IAM, CMKs |
| Training dataset content | — | Lawful collection, classification, validation |
| Model approval for production | — | Verification gates and owners |
| Inference application | Platform hardening | Prompt injection defenses, authZ, logging |
Links to Retention, Auditability, and Earlier Domain 2 Controls
- Retention (2.7): Training snapshots and prompt logs need schedules; legal holds apply to model-related evidence in disputes over automated decisions.
- Auditability (2.8): Log dataset access, training job starts, model downloads, registry changes, and bulk inference exports with identity, IP, and geo attributes.
- Encryption & keys (2.3): Feature stores and model binaries deserve the same key discipline as primary databases.
- Discovery & classification (2.4–2.5): You cannot protect AI data you have not found — shadow notebooks are classic blind spots.
- IRM (2.6): Useful for controlling redistribution of exported model files and sensitive evaluation reports.
Integrated Scenario
A retailer builds a demand-forecasting model on a public-cloud ML platform using three years of transactions plus customer support chat exports. Failures to avoid: dumping full chat (including payment references) into the lake without minimization; training roles with read * on all buckets; no hash manifest for the training set; promoting a model because “loss improved” without security evaluation; logging every inference request with full cardholder-adjacent text for 90 days in an unencrypted analytics project.
Corrected design: classify and tokenize sensitive fields before feature engineering; validate schemas and scan for secrets; verify dataset and model signatures in a registry with dual approval; encrypt stores with CMKs; restrict training subnets from general internet; retain prompt logs per policy with DLP; audit every model download; delete superseded training extracts under retention unless a hold applies.
Exam Approach and Common Traps
- Identify whether the stem is about privacy (leakage, minimization, residency) or security (poisoning, integrity, validation/verification).
- Name the asset: raw data, features, model weights, embeddings, prompts.
- Apply cloud shared responsibility for managed ML services.
- Reuse Domain 2 controls — do not invent a separate universe where “AI magic” replaces IAM and encryption.
- For promotion decisions, look for verification evidence, not only accuracy anecdotes.
Common traps:
- Assuming deleting raw rows removes all sensitive signal from models and embeddings.
- Treating public scraped data as “free of security risk.”
- Skipping data-plane logging on feature stores and model registries.
- Sending regulated data to consumer-grade model APIs without contractual and technical safeguards.
- Confusing validation (fitness/quality/policy checks) with verification (proof properties hold).
Domain 2.9 on the August 2026 CCSP outline most directly requires candidates to comprehend which pair of concerns for AI/ML?
A team plans to train on customer support transcripts that may contain account numbers. Which privacy-oriented control is most appropriate before training begins?
In AI/ML data protection, which activity best illustrates verification rather than validation alone?
Which statement best reflects shared responsibility for a managed cloud ML training service?