3.5 AI Risk Assessment Methodology (ISO/IEC 23894 & Clause 6.1.2)

3.5 AI Risk Assessment Methodology (ISO/IEC 23894 & Clause 6.1.2)

Under ISO/IEC 42001 Clause 6.1.2, organizations must establish, implement, and maintain a formal AI Risk Assessment Process. Standard IT risk management frameworks (e.g., traditional ISO 27005 or NIST SP 800-30) are insufficient for AI because machine learning systems exhibit unique characteristics—such as probabilistic outputs, opaque decision paths, dynamic learning from new data, and vulnerability to data contamination. To address these nuances, Clause 6.1.2 directly aligns with the specialized guidance of ISO/IEC 23894 (Artificial Intelligence — Risk Management).

---

Unique Attributes of AI Risks vs. Traditional IT Security Risks

Understanding the fundamental differences between traditional software systems and AI systems is essential for designing a valid risk methodology:

• Non-Deterministic Behavior: Traditional software produces deterministic outputs based on explicit, hardcoded logic. Machine learning models produce probabilistic predictions that can vary unexpectedly when exposed to novel input distributions.
• Opacity & "Black-Box" Complexity: Deep neural networks often lack explainability, making it difficult to trace why a specific decision was reached or detect latent vulnerabilities prior to failure.
• Data Dependency & Poisoning: AI performance is intrinsically bound to training data quality. Ingesting biased, contaminated, or malicious data directly degrades model integrity.
• Model Drift: Unlike static code, AI model accuracy naturally degrades over time as real-world data patterns shift away from training distribution baselines (concept drift and covariate shift).
• Novel Adversarial Vectors: AI systems face unique attack mechanisms, including prompt injection, evasion attacks (adversarial perturbations), model extraction/inversion, and training data poisoning.

---

AI Threat Modeling & Specific AI Failure Modes

A comprehensive AI risk methodology must perform threat modeling across six specific AI failure modes:

+-------------------------------------------------------------------------+
|                        SPECIFIC AI THREAT VECTORS                       |
|                                                                         |
|  1. Prompt Injection (Direct/Indirect)  4. Model Hallucination & Toxicity|
|  2. Training Data Poisoning             5. Model Inversion & Data Leakage|
|  3. Adversarial Evasion Attacks         6. Concept & Covariate Drift     |
+-------------------------------------------------------------------------+

1. Prompt Injection Attacks (Direct & Indirect)

Malicious users inject adversarial instructions into LLM prompts to bypass system guardrails (direct injection) or embed hidden instructions inside external documents processed by the LLM (indirect injection), causing unauthorized data exfiltration or system takeover.

2. Training Data Poisoning

Attacking the data pipeline by injecting corrupt, biased, or maliciously crafted data samples into training sets, creating backdoors or altering model decision boundaries.

3. Model Hallucination & Toxicity

Generative AI models generating plausible-sounding but completely fabricated, incorrect, or toxic outputs, exposing the firm to severe legal liability and reputational damage.

4. Model Inversion, Extraction & Data Leakage

Reconstructing sensitive training data (including customer PII) or extracting proprietary neural network weights through targeted API queries.

5. Adversarial Evasion Attacks

Applying imperceptible perturbations to input data (e.g., tweaking pixels in an image or subtle text edits) that cause a model to misclassify inputs with high confidence.

6. Model Drift (Concept & Covariate Shift)

Statistical shifts in real-world data distributions over time causing progressive degradation in predictive accuracy and safety.

---

Failure Modes & Effects Analysis (FMEA) for ML Systems

Implementers use Failure Modes & Effects Analysis (FMEA) adapted for machine learning pipelines to systematically evaluate technical failure points:

---

The ISO/IEC 23894 Risk Assessment Lifecycle

ISO/IEC 23894 adapts the standard ISO 31000 risk management framework into a continuous three-stage cycle tailored for AI:

1. Risk Identification: Identifying threat vectors across data pipelines, model artifacts, API endpoints, and user interfaces using FMEA and threat modeling.
2. Risk Analysis: Determining likelihood and consequences across financial, legal, societal, and operational dimensions.
3. Risk Evaluation: Comparing calculated risk scores against the organization's pre-established risk appetite criteria to determine whether risk treatment is required under Clause 6.1.3.

---

Establishing Risk Evaluation Criteria & 5x5 Matrix

Clause 6.1.2 requires documented criteria for evaluating risk levels. Organizations typically employ a 5x5 Likelihood and Consequence Matrix:

• Likelihood Scale (1 to 5): Rare (1), Unlikely (2), Possible (3), Likely (4), Almost Certain (5).
• Consequence Scale (1 to 5): Negligible (1), Minor (2), Moderate (3), Major (4), Catastrophic (5—e.g., severe human rights violation or regulatory shutdown).
• Risk Threshold Matrix:
  • Low Risk (1–5): Acceptable under standard operational oversight.
  • Medium Risk (6–12): Requires documented mitigation controls under Annex A.
  • High / Critical Risk (15–25): Unacceptable; requires immediate executive escalation, risk treatment, or project suspension.

---

Worked Implementation Scenario: Algorithmic Trading Threat Modeling

Context: A hedge fund deploys a deep reinforcement learning model for automated equity trading.

Risk Assessment Execution:

1. Threat Identification: The risk team identifies Covariate Drift (market volatility shifts) and Adversarial Spoofing (market manipulation tactics feeding false signals to the model).
2. Risk Analysis: Covariate Drift is rated Likelihood 4 (Likely) and Consequence 4 (Major trading losses), generating a Risk Score of 16 (High Risk).
3. Evaluation & Treatment Trigger: Because a score of 16 exceeds the fund's risk appetite threshold of 10, the team triggers mandatory Clause 6.1.3 risk treatment, mandating real-time volatility circuit breakers and daily model retraining.

---

Lead Implementer Exam Tips

• ISO/IEC 23894 Relationship: Remember that ISO/IEC 23894 is a guidance standard providing methodology, while ISO/IEC 42001 Clause 6.1.2 is the normative requirement that auditors verify.
• Non-Determinism Focus: Be prepared to explain why traditional static code security reviews fail for AI—specifically due to probabilistic outputs and non-deterministic behavior.
• Prompt Injection Vectors: Distinguish between direct prompt injection (jailbreaking) and indirect prompt injection (embedding malicious payloads in external data sources).