6.3 Gathering and Verifying Objective Evidence (Sampling and Tracing)

Collecting the Proof: Objective Evidence

The fundamental premise of auditing is that conclusions must be based on facts, not assumptions. These facts are referred to as 'objective evidence'. ISO 19011 defines objective evidence as data supporting the existence or verity of something. For an ISMS audit, this evidence proves whether the organization is conforming to the ISO/IEC 27001 requirements and its own policies. The primary methods for gathering this evidence are observation, document/record review, and interviews (as discussed in the previous section). This section focuses on the rigorous methodologies required to collect, sample, and trace this evidence effectively.

Types of Objective Evidence

Auditors should deliberately mix evidence sources rather than relying on a single method. The table below summarizes the main evidence types used in ISMS audits and how each contributes to a defensible finding.

Observation and Records Review

While interviews tell the auditor what the auditee believes is happening, observation and records review provide the concrete proof of what has actually occurred.

Observation: This involves watching a process or procedure being performed. It is particularly valuable for assessing physical security controls (e.g., observing if the reception desk verifies ID badges, checking if server room doors are properly secured, or watching an operator follow a secure disposal procedure). Observation provides high-confidence evidence because it is firsthand. However, the auditor must be aware of the Hawthorne Effect—people tend to behave differently (usually more strictly adhering to rules) when they know they are being observed. Therefore, observation should be corroborated with other evidence types.

Records Review: Records are the historical footprint of the ISMS. While a policy (a document) states what should be done, a record proves what was done. Reviewing records is the most common method of evidence gathering. Examples include firewall configuration change logs, incident response reports, backup success/failure logs, and training attendance sheets. When reviewing records, the auditor must verify several attributes:

• Authenticity — is the record genuine and from the expected system of record?
• Accuracy — does it reflect the reality of the event?
• Completeness — are required fields filled out and properly authorized?
• Timeliness — was the record created when the activity occurred, not reconstructed later?

The Science and Art of Sampling

In any organization larger than a micro-enterprise, it is impossible for an auditor to examine every single record, log entry, or access request generated during the audit period. Therefore, the auditor must take a sample. The goal of sampling is to select a subset of records that is representative of the whole population, allowing the auditor to draw a valid conclusion about the entire system.

Practical sampling steps for an ISMS auditor:

1. Define the population clearly (for example, all privileged access requests raised in the last 12 months).
2. Identify risk factors that should bias or stratify the sample (high-privilege roles, new systems, periods of staff turnover).
3. Choose judgmental or statistical sampling based on population size, data quality, and needed precision.
4. Select the sample items and record the selection rationale in working papers.
5. Test each item against the audit criteria and document conformity or nonconformity.
6. If a nonconformity appears, expand the sample to decide whether the issue is isolated or systemic.
7. Conclude on the control and document residual sampling risk.

There are two primary approaches to sampling in auditing: judgmental and statistical.

Judgmental (Risk-Based) Sampling: This is the most common approach in management system auditing. The auditor uses their professional judgment and experience to select samples based on perceived risk. Instead of randomly selecting 10 out of 100 access requests, the auditor might deliberately select requests related to high-privilege accounts, requests made during a period of high staff turnover, or requests related to a newly implemented system. Judgmental sampling focuses the audit effort where failures are most likely to occur or where they would have the most significant impact. The drawback is that it relies heavily on the auditor's competence and is prone to bias.

Statistical Sampling: This method relies on mathematical principles to select a truly random sample. Every item in the population has an equal chance of being selected. This approach is objective and allows the auditor to quantify the sampling risk (the risk that the sample does not accurately represent the population). However, statistical sampling can be time-consuming to set up, requires a well-defined and accessible population of records, and may result in examining many low-risk records while missing critical anomalies. While less common in general ISMS audits, it may be used when absolute precision is required, such as auditing financial transaction logs or large-scale access control databases.

Regardless of the method chosen, the auditor must document the sampling rationale, the population size, and the specific items selected. If a nonconformity is found within the sample, the auditor must consider whether it is an isolated incident or indicative of a systemic failure, often requiring the sample size to be expanded to determine the extent of the issue.

Computer-Assisted Audit Techniques (CAATs)

In highly digitized ISMS environments, traditional manual sampling of records is often insufficient. Auditors increasingly rely on Computer-Assisted Audit Techniques (CAATs) to analyze large datasets. CAATs can range from simple spreadsheet functions to sophisticated data analytics software. For example, instead of manually reviewing a sample of 50 access logs, an auditor might use a CAAT script to analyze the entire population of 10,000 logs, searching for specific anomalies such as logins during non-business hours, multiple failed authentication attempts, or segregation of duties violations. While CAATs allow for 100% population testing and significantly increase audit assurance, they require specialized technical skills and a thorough understanding of the underlying data structures.

Triangulation of Evidence

Relying on a single piece or type of evidence can be risky. Best practice in auditing involves the triangulation of evidence—corroborating findings by gathering data from multiple, independent sources. For instance, if an auditee states during an interview that all servers are backed up daily (Source 1: Interview), the auditor should not accept this at face value. They should then review the backup policy and schedule (Source 2: Document) and finally examine the system logs from the previous night to verify the backup successfully completed (Source 3: Record). When multiple, distinct pieces of evidence point to the same conclusion, the auditor's confidence in that finding is significantly enhanced. Conversely, if the sources contradict each other, it signals an area requiring deeper investigation to uncover the root cause of the discrepancy.

Tracing Audit Trails

Information security processes rarely exist in isolation; they flow across departments, systems, and personnel. To verify that a process is truly effective, an auditor must follow it from beginning to end, or vice versa. This is known as tracing an audit trail.

Forward Tracing (Tracing): This involves following a process from its initiation to its conclusion to ensure all required steps and controls were applied. For example, to audit the user access provisioning process, the auditor might start with a new employee's offer letter in HR (the trigger). The auditor then traces forward: Was an access request ticket generated? Was it approved by the correct manager? Did IT provision the accounts according to the role-based access matrix? Is there a record of the employee completing security awareness training before receiving access? This method verifies completeness and adherence to the defined workflow.

Backward Tracing (Vouching): This involves starting with the final output of a process and working backward to verify that it is supported by the necessary authorizations and inputs. For instance, the auditor might randomly select an active user account in the Active Directory (the output). The auditor then traces backward: Can IT produce the approved ticket authorizing this account? Does the ticket match the user's current role? Is the user still employed by the organization according to HR records? This method is highly effective for detecting unauthorized activities, phantom accounts, or bypassed controls.

By masterfully combining observation, meticulous records review, strategic sampling, and rigorous tracing of audit trails, the lead auditor constructs a solid foundation of objective evidence. This evidence ensures that the final audit conclusions are robust, defensible, and accurately reflect the true health of the organization's Information Security Management System.