3.4 Interobserver Agreement (IOA) Methods and Calculations

Interobserver Agreement (IOA) Methods and Calculations

In Applied Behavior Analysis, data believability is established primarily through Interobserver Agreement (IOA). IOA measures the degree to which two independent observers, using the same measurement system to monitor the same client during the same observation window, report identical values. High IOA indicates that the operational definition is clear and objective, observer training is adequate, and data collection is consistent.

Critical Conceptual Distinctions

On the QBA examination, candidates must clearly distinguish IOA from accuracy and reliability:

• IOA vs. Accuracy: Accuracy evaluates whether observed data match the true value of the event (measured using an unrefutable gold-standard criterion, such as a calibrated video recording). IOA evaluates agreement between observers. Two observers can achieve 100% IOA while being 100% inaccurate if both misapply an operational definition identically.
• IOA vs. Reliability: Reliability refers to the stability of measurement over time (e.g., whether a single observer scores a permanent product identically on two separate occasions).
• Observer Drift: The gradual, unintended shift in how an observer applies an operational definition over time. Regular IOA assessments detect and correct observer drift.

Professional Standards for IOA

• Frequency: IOA data must be collected across a minimum of 20% to 30% of sessions in each experimental phase.
• Threshold: Acceptable IOA is generally 80% or higher (or a Cohen's Kappa coefficient of 0.60+).
• Independence: Observers must collect data independently (without cues, communication, or looking at each other's data sheets).

---

IOA Calculation Formulas and Worked Examples

1. Count / Event IOA Algorithms

A. Total Count IOA

Calculates agreement based on the total count recorded across an entire session.

$\text{Total Count IOA} = \left( \frac{\text{Smaller Count}}{\text{Larger Count}} \right) \times 100$

• Worked Example: Observer 1 tallies 15 instances of hand-raising; Observer 2 tallies 20 instances.

$\text{Total Count IOA} = \left( \frac{15}{20} \right) \times 100 = 75.0\%$

• Limitation: Overestimates true agreement because it does not confirm observers agreed on specific instances in time.

B. Mean Count-per-Interval IOA

Calculates the average agreement across individual intervals within a session.

$\text{Mean Count-per-Interval IOA} = \frac{\sum \left( \frac{\text{Smaller Interval Count}}{\text{Larger Interval Count}} \right)}{\text{Total Number of Intervals}} \times 100$

• Worked Example: Across 3 intervals: Interval 1 (Obs A = 2, Obs B = 2; Agreement = 1.0); Interval 2 (Obs A = 1, Obs B = 3; Agreement = 0.33); Interval 3 (Obs A = 4, Obs B = 5; Agreement = 0.80).

$\text{Mean Count-per-Interval IOA} = \frac{1.0 + 0.33 + 0.80}{3} \times 100 = \frac{2.13}{3} \times 100 = 71.0\%$

C. Exact Count-per-Interval IOA

The most conservative count metric. Only intervals with 100% exact agreement are scored as agreements.

$\text{Exact Count-per-Interval IOA} = \left( \frac{\text{Number of Intervals with 100\% Agreement}}{\text{Total Number of Intervals}} \right) \times 100$

• Worked Example (using the 3 intervals above): Only Interval 1 had 100% agreement (1 out of 3 intervals).

$\text{Exact Count-per-Interval IOA} = \left( \frac{1}{3} \right) \times 100 = 33.3\%$

---

2. Duration IOA Algorithms

A. Total Duration IOA

$\text{Total Duration IOA} = \left( \frac{\text{Shorter Duration}}{\text{Longer Duration}} \right) \times 100$

• Worked Example: Obs A records 45 minutes of crying; Obs B records 50 minutes.

$\text{Total Duration IOA} = \left( \frac{45}{50} \right) \times 100 = 90.0\%$

B. Mean Duration-per-Occurrence IOA

Calculates duration agreement for each discrete occurrence of behavior and averages them across occurrences.

$\text{Mean Duration-per-Occurrence IOA} = \frac{\sum \left( \frac{\text{Shorter Duration}}{\text{Longer Duration}} \right)}{\text{Total Number of Responses}} \times 100$

---

3. Interval Sampling IOA Algorithms

Interval IOA algorithms are evaluated using a 2x2 contingency matrix of interval scores:

A. Interval-by-Interval (Total) IOA

Includes all intervals (both agreed occurrences and agreed nonoccurrences).

$\text{Interval-by-Interval IOA} = \left( \frac{\text{Agreed Intervals [Both + or Both -]}}{\text{Total Intervals}} \right) \times 100$

• Limitation: Artificially inflates agreement for very high-rate or very low-rate behaviors due to chance agreement on nonoccurrences or occurrences.

B. Scored-Interval (Occurrence) IOA

Calculates agreement based ONLY on intervals where AT LEAST ONE observer scored an occurrence (+). Ignores intervals where both observers scored nonoccurrence (-).

$\text{Scored-Interval IOA} = \left( \frac{\text{Agreed Occurrence Intervals [Both +]}}{\text{Intervals where AT LEAST ONE observer scored +}} \right) \times 100$

• Mandatory Clinical Rule: Scored-Interval IOA must be used for LOW-RATE behaviors (occurring in <30% of intervals) to prevent agreements on nonoccurrences from masking low agreement on actual occurrences.

C. Unscored-Interval (Nonoccurrence) IOA

Calculates agreement based ONLY on intervals where AT LEAST ONE observer scored a nonoccurrence (-). Ignores intervals where both observers scored occurrence (+).

$\text{Unscored-Interval IOA} = \left( \frac{\text{Agreed Nonoccurrence Intervals [Both -]}}{\text{Intervals where AT LEAST ONE observer scored -}} \right) \times 100$

• Mandatory Clinical Rule: Unscored-Interval IOA must be used for HIGH-RATE behaviors (occurring in >70% of intervals) to prevent agreements on occurrences from masking low agreement on nonoccurrences.