7.5 Interobserver Agreement (IOA), Reliability, and Validity

7.5 Interobserver Agreement (IOA), Reliability, and Validity

In Applied Behavior Analysis (ABA), data-driven decision-making depends fundamentally on the trustworthiness of data. If data collection is inaccurate or inconsistent, clinical decisions—such as modifying a Behavior Intervention Plan (BIP) or advancing a skill acquisition target—will be flawed. To guarantee data integrity, behavior analysts evaluate three core measurement pillars: Validity, Reliability, and Accuracy, with Interobserver Agreement (IOA) serving as the primary operational tool for evaluating measurement consistency. Applied Behavior Analysis Technicians (ABATs) frequently participate in IOA assessments alongside supervising BCBAs/QBAs or peer technicians.

---

1. The Three Pillars of Data Integrity: Validity, Reliability, and Accuracy

Validity

Validity exists when a data collection system directly measures the socially significant target behavior that is the actual object of interest, using a quantitative dimension that directly matches the clinical question.

• Valid: Measuring duration when the goal is to increase sitting endurance.
• Invalid: Measuring count of vocal outbursts when the actual clinical concern is how long the child screams (duration).

Reliability

Reliability refers to the consistency or repeatability of a measurement system. A measurement procedure is reliable if it yields the exact same numerical result when repeatedly applied to the exact same physical event across time.

• Note: A measurement system can be reliable without being valid. For example, a faulty scale that consistently adds 5 pounds is reliable (consistent), but invalid (inaccurate).

Accuracy

Accuracy exists when the observed value (the data recorded by the technician) matches the true value of the behavioral event as determined by an undisputed, objective reference standard (e.g., calibrated video analysis).

---

2. What is Interobserver Agreement (IOA)?

Interobserver Agreement (IOA) is defined as the degree to which two independent observers, simultaneously observing the same client during the exact same observation period, record identical data values using the same measurement system.

Primary Functions of IOA in ABA

1. Evaluates Operational Definition Clarity: High IOA confirms that the target behavior definition is objective, clear, and complete enough for different people to score identically.
2. Detects Observer Drift: Over time, observers may gradually shift their interpretation of a behavior definition. Regular IOA checks correct observer drift.
3. Assesses Observer Competency: IOA validates that new ABATs are implementing data protocols correctly.
4. Increases Scientific Credibility: High IOA assures funding sources, families, and researchers that reported behavioral changes reflect actual client progress rather than observer bias.

Minimum Industry Standards for IOA

• Frequency: IOA should be assessed in a minimum of 20% to 30% of sessions across all experimental phases (baseline and intervention).
• Acceptable Score: The minimum acceptable IOA agreement threshold is 80% (or higher).

---

3. IOA Calculation Methods and Formulas

Depending on whether data are continuous or discontinuous, behavior analysts utilize specific mathematical formulas to calculate IOA:

1. Total Count IOA (Continuous Data)

Used for event recording (frequency). Calculated by dividing the smaller total count recorded by one observer by the larger total count recorded by the second observer, multiplied by 100.

$\text{Total Count IOA} = \left( \frac{\text{Smaller Count}}{\text{Larger Count}} \right) \times 100$

• Example: During a 30-minute session, Observer A counts 12 instances of aggression; Observer B counts 15 instances. $\text{Total Count IOA} = \left( \frac{12}{15} \right) \times 100 = 80\%$

2. Mean Count-Per-Interval IOA (Continuous Data)

Divide session into intervals, calculate Total Count IOA for each individual interval, sum the percentages, and divide by the total number of intervals ($N$).

$\text{Mean Count-Per-Interval IOA} = \frac{\text{Interval 1 IOA} + \text{Interval 2 IOA} + \dots + \text{Interval } N \text{ IOA}}{N} \times 100$

3. Exact Count-Per-Interval IOA (Continuous Data)

The most conservative continuous IOA method. Only intervals where both observers recorded the exact same number are scored as agreement.

$\text{Exact Count IOA} = \left( \frac{\text{Number of Intervals with 100\% Exact Agreement}}{\text{Total Number of Intervals}} \right) \times 100$

• Example: Out of 10 intervals, both observers recorded identical counts in 7 intervals. $\text{Exact Count IOA} = \left( \frac{7}{10} \right) \times 100 = 70\%$

4. Total Duration IOA (Continuous Data)

Used for duration measurement. Divide the shorter total duration recorded by the longer total duration recorded, multiplied by 100.

$\text{Total Duration IOA} = \left( \frac{\text{Shorter Duration}}{\text{Longer Duration}} \right) \times 100$

5. Interval-by-Interval (Point-by-Point) IOA (Discontinuous Data)

Used for partial-interval, whole-interval, and momentary time sampling. Compares agreement interval by interval.

$\text{Interval-by-Interval IOA} = \left( \frac{\text{Number of Intervals Agreed (Both } + \text{ or Both } - \text{)}}{\text{Number of Intervals Agreed} + \text{Number of Intervals Disagreed}} \right) \times 100$

• Example: Across 20 intervals, Observer A and Observer B agree on 16 intervals (both scored + or both scored -) and disagree on 4 intervals. $\text{Interval-by-Interval IOA} = \left( \frac{16}{16 + 4} \right) \times 100 = \left( \frac{16}{20} \right) \times 100 = 80\%$

---

4. IOA Methods Comparison Table

The table below summarizes the key IOA formulas, recommended applications, and stringency levels:

---

5. Procedural Rules for Conducting IOA Checks

To ensure IOA assessments are valid, ABATs must adhere to strict procedural rules during IOA sessions:

1. Independent Observation: Observers must sit or stand so they cannot see each other's data sheets, timers, or screens. Neither observer may prompt, signal, or communicate during the observation.
2. Identical Measurement System: Both observers must use the exact same operational definition, measurement metric, and timing device.
3. Simultaneous Timing: Both observers must start and stop observation timers at the exact same second.