Conduct Security, Safety, and Ethics Risk Assessments

Risk Work Belongs at the Front of the Project

Domain II Task 3 is conduct risk assessment(s) (e.g., security, safety, ethics), and its placement matters as much as its content. It sits in Phase 1, Business Understanding, beside problem identification and feasibility — not after a model exists. The reason is structural. In deterministic software, risk is largely about defects, integration, and schedule. An AI system is probabilistic: it will be wrong a knowable percentage of the time, it can be attacked through its data as well as its code, and its errors do not distribute evenly across the people it acts on. Those properties change what the solution has to look like, so they must be known before the solution is drafted. Assess in Phase 1, then revisit the register at every phase gate as data and model behavior become known.

Identify Potential Failure Modes and Safety Implications

Start with the question deterministic projects never ask: what does being wrong cost, and who pays? Analyze false positives and false negatives separately, because they land on different parties.

• Fraud model, false positive: a legitimate customer's card is declined at checkout. The customer bears it; the business sees churn and complaint volume.
• Fraud model, false negative: the fraudulent payment clears. The business bears it as loss and possible regulatory reporting.
• Predictive maintenance, false negative: the asset fails in service. On a conveyor that is downtime; on a brake assembly or an infusion pump it is human harm.

Safety escalates the entire assessment whenever the model touches physical systems (vehicles, plant equipment, robotics) or human welfare (clinical, benefits, hiring, credit, safeguarding). For those cases, define hard operating limits, a fail-safe default state, and a route by which a person can override or stop the system — and record them as requirements now, because retrofitting them after Phase 4 is expensive and often architecturally impossible.

Assess Cybersecurity Vulnerabilities in AI Systems

A CPMAI project manager is not expected to implement defenses, but must be able to name the AI-specific attack surfaces and require that security review covers them, because a conventional application security review will not. Published threat catalogs such as MITRE ATLAS enumerate these in depth; you need the vocabulary and the judgment, not the implementation.

• Data poisoning — corrupted or crafted records injected into training or retraining data so the model learns wrong behavior or a hidden trigger. Highest where data is scraped, crowdsourced, or continuously retrained on user input.
• Model evasion (adversarial input) — inputs perturbed at inference time so a working model misclassifies them. The attacker never touches your model or your data.
• Model inversion and membership inference — repeated querying used to reconstruct sensitive training records, or to confirm that a named individual was in the training set. This makes a model itself a privacy exposure even though the underlying data never left.
• Model theft (extraction) — systematic querying of an exposed prediction interface to reproduce a proprietary model's behavior, losing the asset you paid to build.
• Prompt injection — instructions hidden in user input or in retrieved content (a document, a web page, an email) that override a generative system's rules, exfiltrate data, or trigger unauthorized actions.
• Supply-chain risk — pre-trained models, open weights, third-party datasets, and vendor application programming interfaces (APIs) inherit whatever was in them. Require provenance, licensing terms, and update and deprecation terms for anything you did not build.

Evaluate Ethical Implications of AI Decision-Making

Ask three questions of every automated decision: who is affected, do they know, and what recourse do they have? Then ask whether error rates differ across groups — a strong aggregate accuracy figure can conceal a model that performs materially worse for one population. Domain I covers bias checks and transparency in depth; your Phase 1 job is to locate the ethical exposure and record it, so that explainability, appeal routes, and oversight enter the scope statement rather than arriving as a late and unbudgeted surprise.

Analyze Reputational and Business Continuity Risks

Reputational risk asks what a single bad output looks like when it is screenshotted, published, or quoted in a regulator's letter. For generative systems, one confidently wrong answer given to a customer is a publishable event, and the volume of outputs makes such an event a near-certainty over time. Continuity risk asks what happens when the system is unavailable or degraded: does the manual process still exist, do staff still have the skill to run it, and what breaks if a third-party API suffers an outage, reprices, or retires the model version you built on? Vendor model deprecation runs on the vendor's schedule, not yours, and belongs in the register from day one.

Develop Risk Mitigation Strategies and Contingency Plans

Convert all of the above into a risk register in which each entry carries a description, a category, likelihood and impact, an owner with authority to act, an early control, a trigger — the observable condition saying the risk is materializing — and a contingency. Triggers are what make an AI risk register operational rather than decorative: "accuracy for the over-65 segment falls below the agreed floor," "manual override rate exceeds 20%," "vendor announces deprecation of the model version in use."

Severity Drives the Level of Human Oversight

The judgment the exam most wants from you is this: the severity and reversibility of harm determine how much human oversight is designed into the solution. A low-severity, easily reversed decision — which article to recommend next — can run fully automated. A high-severity, hard-to-reverse decision — deny a claim, take a medical device out of service, halt a production line — requires human-in-the-loop approval, decision logging, and an appeal or override route. That choice changes latency, staffing, and cost, which is precisely why it is settled here in Phase 1, before scope and return on investment are fixed, and why a risk register that is written once and never reopened is a defect rather than a deliverable.