4.2 AI and Safety

What safety-related means in this chapter

A safety-related system has the potential to cause injury or harm to people, property, or the environment. Developing and testing non-AI safety-related systems already takes a lot of effort, but the pattern is known: specify, refine, implement, trace, verify, and argue residual risk. For AI-based systems there are several additional challenges. CT-AI v2.0 learning objective AI-2.1.2 (K2) asks you to explain those special considerations.

Safety is not the same label as ISO/IEC 25059 AI robustness or intervenability, though those product qualities often support a safety argument. Safety is about harm potential. A chatbot that is merely rude is usually not a safety-related system. A medical-device inference path that can withhold a critical alarm is.

Independent OpenExamPrep teaching here uses the syllabus challenge list as the exam map. Functional-safety standards and the EU AI Act are references you should recognize by name and effect; the exam is not a lawyer's reading of every article.

Challenge 1: specifications and inadequate traceability

In traditional safety work, requirements are defined for the complete system and then refined until developers can transform them into code. Traceability from hazard to requirement to design to implementation to test is how you argue that a hazardous function was specified and checked.

Many AI-based systems do not start that way. They start from vague goals—detect pedestrians, triage the queue, keep the plant in a safe envelope—that are then implicitly provided via the training data. The data encodes patterns, rules, and objectives without fully formalizing every detail up front. The result is that traceability from requirements to implementation is often inadequate for the level of assurance safety programs expect.

Testing implication. You cannot treat trained weights as if they were source lines traced to numbered SHALL statements. Testers push for an explicit operational design domain, labeled failure modes, data-requirement specifications, and tests that probe what the goal document never stated. If a pedestrian-at-night case was never written down and barely appeared in the data, the safety argument has a specification hole, not just a model-accuracy hole.

Worked example. An automotive emergency-braking function is described as reduce collisions with vulnerable road users. The training set is mostly daytime urban clips. A night-time cyclist with a weak reflector is not a missed unit test of a specified branch. It is evidence that the goal was never refined into complete, testable requirements. Traceability to the implementation (the model) is weak because there was no refined requirement to trace.

Challenge 2: non-determinism

Many AI-based systems are non-deterministic. Testers cannot guarantee the precise behavior of these systems. Even rigorously tested models can exhibit unexpected behavior because of random number generation or slight variations in input values.

Randomness shows up in more than one place: dropout left on at inference, stochastic sampling in generative models, shuffled data loaders during online updates, or non-associative floating-point reductions on a GPU. Slight input variation is just as common: sensor noise, compression artifacts, or a one-bit change in a preprocessing buffer.

Testing implication. Repeatability has to be designed. Fix seeds where the architecture allows, record input hashes, define tolerance bands instead of exact-output oracles, and treat a flaky safety-relevant result as a finding, not as a lab nuisance. A braking-distance estimate that jitters by tens of centimetres on identical logged frames is a non-determinism finding.

Worked example. Two replays of the same lidar frame differ by one bit in a preprocessing buffer. The detector drops a pedestrian box on one replay and keeps it on the other. Traditional safety evidence that test T-17 passed is incomplete unless you state how many replays, what variance is allowed, and what you do when the model disagrees with itself.

Challenge 3: self-learning after deployment

Before deployment, rigorous testing is used to demonstrate safety integrity for a frozen behavior. For self-learning AI-based systems, that demonstration is undermined because the system's behavior progressively moves away from the originally tested behavior.

Two syllabus-level responses:

• Manage how the model learns and the data it uses, so new problematic behaviors are less likely to emerge. That means curated update pipelines, hold-out safety sets, and human approval of data that is allowed to change the model.
• Safety guards that prevent the model from learning or deciding in ways that compromise safety. A concrete example is a content-moderation component that filters prompts before a generative model can act on them.

Worked example. A warehouse robot is signed off on a fixed policy. Production then enables on-robot learning from near-miss logs. After two weeks the robot takes a shortcut across a pedestrian aisle that never appeared in the test suite. The original tests are not wrong. The system is no longer the system you tested. Either lock learning, constrain the data that can update the policy, or wrap the learner with a guard that forbids aisle-crossing actions regardless of the latest weights.

Self-learning is a process risk as much as a model risk. Configuration that silently turns on online updates is a testable item. So is the presence or absence of a guard that cannot be bypassed by the learner.

Challenge 4: explainability and transparency

For safety-related systems it is essential to understand how and why the system makes decisions. The decision-making processes of AI-based systems are often not transparent. Explainable AI (XAI) techniques can provide insight into the system's reasoning. The syllabus names LIME (Local Interpretable Model-agnostic Explanations) as an example technique: it explains an individual prediction by probing the model locally with perturbed inputs.

Limits you must be able to state:

• These techniques are not widely available in many industrial safety toolchains.
• They may compromise system performance (extra inference, approximations, or constraints on model class).

Transparency as an ISO/IEC 25059 characteristic—communicating appropriate information to stakeholders—is related but not identical. A version string on a dashboard is transparency. A LIME map over a radiograph is an XAI technique that may support a safety explanation. Neither automatically proves the system is safe. An explanation can be plausible and still be wrong about the model's true dependence on a feature.

Worked example. A nuclear-plant advisory model recommends a valve change. Operators cannot see which sensor combination drove the recommendation. Adding LIME in a shadow mode helps a safety case only if the explanations are themselves validated, if latency remains acceptable, and if operators are trained not to treat a heatmap as a proof of correctness.

Challenge 5: evolving regulations

The regulatory landscape for safety-related AI-based systems is constantly evolving. The use of AI is currently not included in mature functional-safety international standards, and some of those standards even prohibit its use in such systems. You cannot assume that following a familiar functional-safety programme automatically covers a learned component.

The EU AI Act classifies AI systems used as safety components—examples include aviation, medical devices, and automotive—as high-risk and imposes strict requirements on their development and testing. High-risk status is a regulatory classification with duties. It is not a substitute for a hazard analysis, but it tells testers that evidence packs, data governance, and post-market monitoring will be scrutinized.

Worked example. An AI inference path is inserted into a steering-support function. The conventional, non-AI portions of the steering stack still have to satisfy the applicable automotive functional-safety work (for example ISO 26262-6 at a stated ASIL). The AI portion cannot hide inside ordinary software unit tests if the Act treats it as a high-risk safety component. Some functional-safety standards may still restrict or forbid that AI use until the organization has an approved argument.

How the five challenges stack for testers

The challenges are not five isolated trivia facts. They compound.

Specification gaps make traceability weak. Non-determinism makes a single passing run a weak oracle. Self-learning invalidates yesterday's evidence. Weak explainability makes incident investigation slow. Regulation may demand more evidence than the project planned. A practical test strategy therefore combines frozen-model campaigns, replay at defined seeds, operational monitoring, learning-control or guards, and an explicit map from hazards to tests.

Acceptance criteria for safety itself are taught in the next section. This section is the why-it-is-harder argument you must be able to explain in your own words.

Exam classification reminders

• Harm to people, property, or environment → safety-related context.
• Vague goals plus training data as the real specification → specification and traceability challenge.
• Random-number generation or tiny input change → non-determinism.
• Post-deployment learning that leaves the tested behavior → self-learning; look for data management or safety guards.
• LIME or similar, with availability and performance caveats → explainability.
• Standards silent or prohibitive; EU AI Act high-risk safety components in aviation, medical, or automotive → evolving regulation.

If a stem gives you a medical infusion pump whose dose model updates from ward data overnight, you should be able to name self-learning, the specification gap in the original dose policy, and the regulatory pressure on a medical safety component—without claiming you have certified the device.