14.3 Monitoring, Reviewing & Improving the Program

14.3 Monitoring, Reviewing & Improving the Program

Establishing an AIMS audit programme is the Plan phase; running audits is Do. ISO 19011 Clause 5 also requires the programme to be monitored, reviewed, and improved. Without Check–Act, the programme drifts: schedules slip, the same Annex A weaknesses recur, and competence falls behind new generative-AI practices.

This section is about managing the programme, not about Clause 9.1 monitoring of the AIMS itself—though programme outputs feed AIMS management review.

---

Why programme monitoring matters for AI

AI estates change faster than many traditional management systems. A programme that was adequate when the organization had three classical ML models may be inadequate after multi-region LLM deployment. Monitoring answers:

• Are we executing the audits we committed to?
• Are audits producing reliable, timely insight on high-risk AI?
• Are auditors performing and remaining competent?
• Are findings closed and verified, or stacking up?
• Is the programme still aligned with risk, scope, and stakeholder needs?

---

KPIs for an AIMS audit programme

Select a balanced set—delivery, quality, coverage, and learning. Avoid vanity metrics (pages of report) that ignore assurance value.

Example quarterly dashboard: plan attainment 83% (target ≥90%); high-risk AI sampled 12/15; majors overdue verification 3; audits with full competence 7/9; mean auditee feedback 4.1/5.

Exam trap: High plan attainment with zero sampling of high-risk AI systems is not success. Coverage and risk alignment matter as much as calendar compliance.

Using KPIs without weaponizing them

• Investigate root causes (resource cuts, access denial, poor scoping) before blaming auditors alone
• Do not reward “zero NCs” as a KPI—that creates under-reporting pressure and violates integrity/fair presentation culture
• Compare like with like: Stage 1 readiness audits vs deep Stage 2 implementation audits differ

---

Monitoring activities (ongoing Check)

Programme managers should routinely:

1. Track schedule vs actual and reforecast remaining year capacity
2. Review NC and opportunity logs across audits for systemic themes
3. Sample audit reports/working papers for consistency with ISO 19011 principles and AIMS criteria
4. Verify that team appointments matched the competence matrix
5. Monitor impartiality disclosures and complaint handling
6. Confirm follow-up audits/verifications are scheduled, not optional
7. Watch external signals: major product launches, incidents, regulatory inquiries, M&A—triggers for programme adjustment

For CBs, monitoring also includes scheme-level consistency, auditor calibration, and accreditation body expectations under ISO/IEC 17021-1.

---

Management review of the audit programme

Periodically (e.g., annual, with interim updates), top management (internal programme) or CB governance reviews the programme. This is distinct from—but inputs to—the organization’s AIMS Clause 9.3 management review.

Typical programme review inputs

• KPI results and trends
• Achievement of programme objectives
• Changes in AI estate size, complexity, risk, and certified/declared scope
• Resource adequacy and competence gaps
• Status of programme-level risks and opportunities
• Feedback from auditees, clients, regulators, accreditation (as applicable)
• Results of external assessments of the audit function/CB
• Follow-up status of previous programme improvement actions

Typical outputs

• Revised objectives and extent (more/fewer audits; new thematic AI focus)
• Resource decisions (hires, training budget, tools)
• Procedure updates (remote audit rules, confidentiality for model IP)
• Schedule changes and special-audit criteria refinements
• Competence development plans
• Decisions on combined multi-standard approaches

AIMS scenario: KPIs show 95% plan attainment but only 40% of high-risk systems sampled because inventory was outdated. Management review expands extent, funds two additional AI-competent auditors, and requires monthly inventory reconciliation with product management—not merely “try harder next year.”

---

Continual improvement of the programme

Improvement is the Act phase: deliberate change based on evidence.

Document improvement actions with owners and due dates. Verify effectiveness in the next monitoring cycle (Did high-risk coverage rise? Did report cycle time fall without quality loss?).

---

Lessons learned from AIMS audits

Cross-audit learning is a programme asset. Common AIMS themes that should drive improvement:

1. Impact assessments exist as templates but not for all high-risk systems → thematic audit + earlier Stage 1 challenge of inventory vs assessments
2. Third-party foundation models weakly controlled (A.10) → add supplier AI audit stream; require contract evidence packages
3. Monitoring metrics not linked to intended use or fairness concerns → update sampling to include metric design interviews, not only dashboard screenshots
4. Confidentiality incidents with model IP in working papers → tighten evidence-handling procedure and tools
5. Combined ISMS auditors missing AI lifecycle depth → mandate AIMS-specific competence before solo leading
6. NCs closed with policy rewrites only → programme rule: effectiveness verification requires operational evidence

Facilitate short lessons-learned sessions after major certification cycles or annually for internal programmes. Capture what to start/stop/continue in the programme plan.

---

Linking programme improvement to organizational improvement

Do not confuse the two loops:

Programme outputs (findings trends, coverage gaps) inform AIMS improvement and management review, but rewriting the auditee’s AI policy is not “improving the audit programme.” Conversely, training auditors is not a substitute for the auditee fixing nonconformities.

---

Practical monitoring calendar

  Monthly:  schedule status, open NC aging, competence conflicts for upcoming audits
  Quarterly: KPI dashboard, risk register refresh, inventory delta review
  After each major audit: file lessons, report QA sample, auditee feedback
  Annually:  full management review of programme; update objectives/extent/resources
  Triggered: special review after serious AI incident, accreditation finding, or mass product launch

---

Exam focus

Domain 7 expects you to name programme KPIs, distinguish programme management review from a single audit’s closing meeting, and show how lessons from AIMS audits change next year’s competence, sampling, and special-audit triggers. Monitoring without improvement is incomplete; improvement without data is guesswork. Together they keep the AIMS audit programme trustworthy as AI systems evolve.