Oversee the AI Solution Contingency Plan

Contingency for Systems That Fail Differently

Conventional continuity planning asks one question: is it up? Artificial intelligence (AI) adds a class of failure that answers yes while causing real harm. ECO Domain V, Task 7 asks you to oversee a contingency plan built for that reality, and the first job is to enumerate what can actually go wrong.

Five failure modes belong in every AI incident procedure:

1. The model is unavailable. Endpoint down, compute exhausted, dependency outage. Conventional, and the easiest to detect.
2. The model is available but silently wrong. It returns confident, well-formed predictions that are no longer accurate. Nothing alerts. This is the defining AI failure mode, and it is why performance monitoring against ground truth, not just uptime monitoring, must be part of the plan.
3. Upstream data stops or changes shape. A feed stalls, a field is renamed, units change, a categorical variable gains a new value. The model keeps scoring, on garbage.
4. The model produces harmful, biased, or offensive output. A conversational assistant says something unacceptable; a scoring model degrades disproportionately for one group. The consequence is reputational and regulatory rather than technical.
5. The model is being abused or attacked. Prompt injection, systematic probing to find the decision boundary, deliberate feeding of crafted inputs, or attempts to extract training data.

Incident Response: Severity and the Keep, Degrade, or Stop Decision

Severity classification for AI should reflect the consequence of a wrong decision, not merely availability. A lending model producing silently biased declines is a higher-severity event than an outage of an internal recommendation widget, even though only the outage appears on the availability dashboard. For each severity level, define the response time, who is engaged, and the single decision that matters:

• Keep serving is defensible when performance is degraded but still above the floor agreed in Business Understanding and confirmed at the Phase 5 readiness gate, and the harm of stopping exceeds the harm of the error.
• Stop entirely is right when output may be harmful or when the model's behavior is not understood.
• Degrade to a fallback is where most real incidents land.

Graceful Degradation, and the Fallback That Must Be Real

Three degradation options, each with a limit you should know before the incident:

• Revert to the previous model version. Fastest, and correct when the fault is in the new version. Useless when the fault is upstream data, because the older model consumes the same broken feed.
• Fall back to a rules engine or the prior manual process. Slower and less accurate, but predictable and explainable.
• Route everything to human review. Safest, and the only acceptable option when outputs may be harmful, but strictly capacity-limited.

The point exams press on: a fallback capable of carrying production volume must be maintained, not merely documented. If the manual process was decommissioned at go-live, the staff redeployed, and the rules engine left untouched for two years, you do not have a fallback — you have a paragraph. Decide what volume the fallback must carry, fund keeping it in that condition, and if human review is the fallback, know how many cases per hour reviewers can genuinely absorb and what the backlog looks like after four hours.

Backup and Recovery for Models

Three artifacts must be recoverable, not one:

• The model artifact plus its exact serving configuration and version, so a known-good version can be restored quickly.
• The training data and labels, plus the code and parameters that produced the model. Losing labeled data is far worse than losing a model: the model can be rebuilt from the data, the data cannot be rebuilt from the model, and re-labeling can cost more than the original project did.
• The pipeline — feature transformation logic and the reference data it depends on. A restored model fed by differently transformed input is not the same system.

Set explicit recovery objectives with the business: a recovery time objective (how long the process can run without the model) and a recovery point objective (how much recent data you can afford to lose). Both must come from the process owner and be tested against the fallback's real capacity, not asserted by the technical team.

Escalation: Who Is Authorized to Pull the Switch

Name three things in advance. First, who can authorize taking a customer-facing model offline — most engineers do not believe they are empowered to do it and most executives cannot be reached fast enough, so pre-authorize a role, usually the accountable service owner or the on-call incident commander, within defined criteria. Second, who must be notified when an incident has regulatory implications — privacy, compliance, and legal, often on a statutory clock where personal data is involved under regimes such as the General Data Protection Regulation (GDPR). Third, who handles reputational exposure, meaning communications and the executive sponsor. Agree all three before an incident, never during one.

Business Continuity and the Decisions Already Made

Business continuity is about the work the model now performs, not the model. If the solution triages 4,000 claims a day and stops, the plan must state what happens to those claims: queued, routed to a reduced-scope rules set, or handled manually at a lower throughput, and for how long that is tolerable to the business.

Then the question most plans omit. If the model was silently wrong for eleven days, decisions were made and acted upon — claims denied, maintenance not scheduled, applications declined, prices set. Restoring service does not end the incident. The procedure must identify the affected population from the prediction log (this is precisely why the audit trail exists), set a review threshold, re-decide or reverse where warranted, notify affected people where required, and record what was done.

Test It, or It Is Only an Assumption

Rehearse on a defined cadence: execute the rollback, fail over to the fallback, and restore from backup. Inject a simulated drift or a broken feed and confirm that a human is actually alerted and knows the next step. Run a tabletop for the harmful-output scenario, because that one is resolved by people rather than automation. Record every test result and feed it back into the runbook. Task 7 names testing and validating contingency procedures as an enabler for one reason: an untested fallback is not a control, it is an assumption.