13.1 Classifying Nonconformities & Observations

13.1 Classifying Nonconformities & Observations

Auditor focus: Classification is a professional judgment under ISO 19011 and certification practice (ISO/IEC 17021-1), not a negotiation with the auditee. Wrong severity distorts certification risk; vague wording makes verification and corrective action impossible.

From evidence to findings

During Stage 2 (and internal or supplier AIMS audits), the team evaluates audit evidence against audit criteria—ISO/IEC 42001 clauses, applicable Annex A controls, the organization’s AIMS documented information, and other agreed requirements (legal, contractual, or client-specific). The output is audit findings: conformity, nonconformity, or other observations.

A nonconformity (NC) is the non-fulfillment of a requirement. Requirements may be:

1. Explicit ISO/IEC 42001 clause obligations (e.g., 6.1.4 AI system impact assessment where applicable)
2. Applicable Annex A controls the organization has determined are needed (and related SoA / control decisions)
3. The organization’s own mandatory AIMS procedures and criteria
4. Other requirements included in the audit criteria (regulation, contracts, customer AI policies)

If there is no requirement, there is no nonconformity—only perhaps an OFI or a note outside criteria.

Major vs minor nonconformity (management-system practice)

Certification bodies and lead auditors commonly use two severity levels. Exact CB procedures may use additional labels, but the conceptual split is stable across MS audits:

Major NC—definition in practice

Treat a finding as major when there is total absence of a required AIMS element; systemic failure of the same requirement across systems/sites/samples; breakdown of intended outcomes (AIMS cannot reasonably govern AI risk/impact); or failure of a critical clause theme that undermines confidence in the system as a whole (e.g., operational control that never gates production models).

AIMS example — major: No AI system impact assessment process, no procedure, no roles, and no completed assessments for any high-impact systems in scope. That is absence of a core planning/control expectation under ISO/IEC 42001—not “documentation polish.” Grade major.

Minor NC—definition in practice

Treat a finding as minor when a requirement is not met but extent is limited, the process exists and usually operates, and residual risk is localized without redesigning the whole AIMS.

AIMS example — minor: Procedure requires a current model card for each production model; 14 of 15 cards are current; one recommender still lists an old evaluation suite and a retired owner after a reorg. Process exists; failure is isolated → minor.

Do not inflate or deflate severity: total process absence is not “minor paperwork”; one missing signature is rarely a major; related minors that show the same root failure mode across samples (e.g., no evaluation artifacts on every high-risk release for six months) may warrant a major systemic finding instead of five separate minors. Grade on evidence—not pressure tactics.

Observations and opportunities for improvement (OFI)

Not every improvement idea is a nonconformity. An observation / OFI applies when criteria are met (or no requirement is failed) but the auditor notes a way to strengthen effectiveness or clarity. Positive findings are optional for balance—not substitutes for NCs.

AIMS example — OFI: Impact assessments and model cards meet criteria, but performance and fairness metrics are documented inconsistently across teams, making management review harder. Recommend a shared metrics taxonomy—OFI, not NC, unless criteria already mandate a specific metrics standard that is missing.

OFI discipline: never use an OFI to avoid a genuine NC; never invent requirements outside criteria; phrase OFIs as voluntary suggestions; track them separately from NCs.

Writing clear findings: three parts

Every nonconformity (and strong observation) should be readable by someone who was not on the audit. Use three elements:

Avoid vague language: “culture is weak,” “AI ethics is immature,” “team seemed confused.” Replace with who, what, when, which requirement, which sample. Confirm the requirement is in criteria, evidence is retrievable, severity matches extent, and wording stays within report confidentiality limits.

AIMS classification scenarios (worked)

Grading discipline and consistency

Before the closing meeting, the lead auditor calibrates the team: same evidence should yield the same grade and consistent extent language (“1 of 5 sampled systems” vs “organization-wide”). Apply the same yardstick to AI and non-AI themes—do not soft-grade human-oversight logging failures for automated credit decisions while hardening unrelated admin gaps. Follow the CB’s classification procedure for certification audits; for internal audits, keep major/minor logic coherent so Stage 2 is not a surprise. Listen to factual corrections (wrong sample ID, superseded procedure); re-evaluate only on new evidence, not relationship pressure.

Classification drives follow-up intensity (Section 13.4) and certification recommendations (Section 13.2): majors need urgent plans and deeper verification; minors may allow longer windows and document checks. Grade on requirement + evidence + systemic extent. Write findings so a stranger can audit your judgment.