12.2 Collecting Evidence: Interviews, Observation, Documents

12.2 Collecting Evidence: Interviews, Observation, Documents

Auditor focus: AIMS findings stand or fall on verifiable evidence. A confident interview without records, a polished policy without operation, or a dashboard screenshot without context is incomplete. Use three primary methods together, then triangulate.

What counts as audit evidence

Per ISO 19011, audit evidence is records, statements of fact, or other information that are relevant to the audit criteria and verifiable. Evidence is evaluated against criteria (ISO/IEC 42001 requirements, applicable Annex A controls, organizational AIMS documented information, and other agreed requirements) to produce findings.

For AI management systems, evidence often mixes:

• Management-system artifacts (policy, risk/impact assessments, objectives, internal audit, management review)
• Technical lifecycle artifacts (model cards, training data docs, evaluation reports, deployment tickets)
• Operational signals (monitoring, human overrides, incidents, drift alerts)
• People evidence (competence, awareness, role understanding, real use practices)

Sufficiency is not “more PDFs.” Sufficiency is enough independent, relevant, reliable information to support a conclusion for the sampled process or system.

Three primary methods

Treat any technical verification as controlled observation/document checks—never unconstrained “hacking” of production AI.

Triangulation (non-negotiable)

Triangulation means corroborating one source with another before treating a claim as strong evidence.

Example: A model owner says “every production release needs fairness evaluation.” Corroborate with:

1. Documented release procedure and evaluation criteria
2. Sample of recent release tickets with attached evaluation reports
3. Observation of the CI/CD or registry gate that blocks unapproved promotion

If interviews say “yes,” procedures say “yes,” but the last five releases lack evaluation artifacts, the triangulated finding points to a performance/implementation gap, not conformity.

Interview techniques by audience

Executives and AIMS leadership

Goal: intent, resources, accountability, and how AI risk reaches governance. Use open → probe → closed questions (assurance routes, management-review inputs, policy approval dates). Listen for resources and decisions, not slogans—“we take AI ethics seriously” is not evidence; minutes that fund remediation after a fairness incident can be.

Model owners and product owners

Goal: lifecycle control reality—intended use, changes, oversight, monitoring ownership.

Ask for a story of one system end-to-end: problem framing → data → training/evaluation → deployment approval → production monitoring → incident/change. Follow the trail with “show me” requests. Challenge mismatches between intended use statements and how sales or operations actually apply the model.

Data engineers and ML platform teams

Goal: data quality, lineage, access control, reproducibility, and pipeline change control.

Prefer concrete artifacts: dataset version IDs, lineage graphs, labeling guidelines, feature store configs, pipeline run IDs linked to a model version. Avoid abstract “we use best practices.” Ask what happens when training data schema drifts or a toxic subset is discovered post-deployment.

End users and human-in-the-loop operators

Goal: whether A.9 use controls and human oversight exist in the field.

Ask how users are told they are interacting with AI (where required), how they escalate wrong outputs, whether overrides are logged, and whether production pressure overrides policy. End-user interviews often surface shadow AI tools or prompt practices that never appear in the official inventory.

Observation of AI operations and monitoring

Observation is more than a facility walkthrough. For AIMS, observe:

• Model registry / deployment console — version lineage, approval status, environment promotion
• Monitoring dashboards — performance, data drift, prediction drift, fairness slices (where claimed), latency/error budgets that affect safe operation
• Human oversight queues — review SLAs, sampling of reviewed cases, escalation paths
• Incident / problem tickets for AI failures — classification, containment, learning
• Change advisory or release meetings if they gate high-risk models

When observing dashboards, note what metrics exist, alert thresholds, owners, and recent alert handling—not only that a pretty UI exists. Ask to open a recent alert and follow it to ticket and action. Observation without follow-through is theater.

For remote observation, have the owner navigate while you direct views; prepared recordings may supplement but should not replace interactive sessions for critical systems.

Document and record review

Distinguish documents (say what should happen) from records (show what did happen).

Typical AIMS document/record sets:

• Scope, AI policy, roles, competence matrices
• Risk assessments, AI system impact assessments, Statements of Applicability-style control decisions if used
• Lifecycle procedures: design, V&V, release, monitoring, retirement
• Model/system cards, data sheets, evaluation reports
• Training/awareness records for AI-related roles
• Internal audit and management review outputs covering AIMS
• Third-party contracts and due diligence for supplier models/APIs

Review for currency, approval, consistency with inventory, and linkage. An impact assessment dated two years before a major use-case change is a red flag. A model card that claims monitoring the organization cannot show live is another.

Recording evidence and avoiding leading questions

Working notes discipline

For each meaningful item, capture:

• Date/time, location (physical/virtual), participants
• Criterion referenced (clause/control/procedure)
• What was said / shown / sampled (IDs, versions, ticket numbers)
• Your evaluation (conformity, potential NC, need more evidence)
• Cross-references to files saved in the working paper pack

Write facts, not adjectives: “Release ticket DEP-4412 for model recsys-v3.2 has no attached fairness report; procedure REL-AI-02 §4.2 requires one for high-impact ranking models” beats “sloppy release process.”

Avoid leading questions

Leading questions invite social compliance and pollute evidence. Use open stems, then probe, then confirm with closed questions only after the interviewee has stated a process.

Cultural and technical communication challenges

AIMS audits cross engineering, business, and language cultures. Manage jargon with plain-language definitions local to the organization; hierarchy by interviewing some practitioners without the full management chain when appropriate; production fear with read-only paths; multi-language teams with interpreters and restated confirmations; and vendor black boxes by shifting to contracts, diligence, output monitoring, and residual risk acceptance. Stay courteously skeptical—restate unclear answers instead of leading or ambushing.

Putting the methods together

For a high-impact system, a practical day pattern is: brief owner interview → pull impact assessment, model card, evaluation, and release procedure → observe registry, monitoring, and one recent release ticket → follow gaps with data/ops interviews → end-of-day triangulation note (sufficient / need more / potential finding). That cadence turns three methods into one coherent evidence story for AI-specific artifacts and formal findings.