13.3 Interobserver Agreement (IOA) Formulas, Methods, & Standards

Interobserver Agreement (IOA) Formulas, Methods, & Standards

Exam Tip: Interobserver Agreement (IOA) calculations appear repeatedly on the QASP-S examination. You must be prepared to compute Total Count IOA, Mean Count-per-Interval IOA, Exact Count-per-Interval IOA, Total Duration IOA, Mean Duration-per-Occurrence IOA, Total Interval IOA, Scored-Interval IOA, and Unscored-Interval IOA using provided clinical datasets. Crucially, memorize the clinical rules for interval IOA: Scored-Interval IOA is mandatory for low-rate behaviors ($< 30%$ of intervals) to avoid artificial inflation from non-occurrences, while Unscored-Interval IOA is mandatory for high-rate behaviors ($> 70%$ of intervals) to avoid artificial inflation from occurrences. Remember the professional QABA standards: IOA collected in 20% to 33% of sessions with an acceptable threshold of $\ge 80%$.

In Applied Behavior Analysis, empirical data drive every supervisory and clinical decision—from determining whether a child has mastered an expressive communication skill to deciding whether a behavior intervention plan (BIP) should be faded or intensified. However, because behavioral data are gathered by human observers (behavior technicians, paraprofessionals, caregivers, and supervisors), the data are inherently susceptible to human measurement error, bias, fatigue, and misunderstanding.

To ensure that behavioral data are trustworthy, reliable, and scientifically robust, behavior analysts rely on Interobserver Agreement (IOA). IOA is defined as the degree to which two or more independent, simultaneous observers report the same observed values after measuring the exact same behavioral events.

---

Scientific & Clinical Functions of IOA

IOA is not merely an academic exercise; it serves four vital clinical and quality-assurance functions in autism service delivery:

                               ┌─────────────────────────────────────────┐
                               │         FOUR CORE FUNCTIONS OF IOA      │
                               └────────────────────┬────────────────────┘
                                                    │
         ┌───────────────────────────┬──────────────┴────────────┬───────────────────────────┐
         ▼                           ▼                           ▼                           ▼
┌─────────────────┐         ┌─────────────────┐         ┌─────────────────┐         ┌─────────────────┐
│   OPERATIONAL   │         │    OBSERVER     │         │    OBSERVER     │         │      DATA       │
│   CLARITY       │         │   COMPETENCE    │         │      DRIFT      │         │  BELIEVABILITY  │
│ Evaluates if    │         │ Assesses if     │         │ Detects if      │         │ Assures funders │
│ the definition  │         │ technicians are │         │ staff criteria  │         │ & families that │
│ is objective,   │         │ fully trained   │         │ shift over time │         │ progress is real│
│ clear & complete│         │ and calibrated. │         │ unconsciously.  │         │ not an illusion.│
└─────────────────┘         └─────────────────┘         └─────────────────┘         └─────────────────┘

Believability vs. Accuracy vs. Validity

The QASP-S exam tests your understanding of the precise epistemic boundaries of IOA:

1. Validity: A measurement system is valid if it measures the actual dimensional property of the behavior that is the true target of intervention. IOA does not assess validity. Two observers can achieve 100% agreement on a completely invalid measurement system (e.g., both measuring child weight with a scale when the clinical target is on-task attention).
2. Accuracy: A measurement system is accurate if the observed values match the true values of the event, determined by independent calibration instruments (e.g., millisecond-calibrated video playback). IOA does not evaluate accuracy. Two observers can share the exact same misunderstanding and achieve 100% agreement while both being completely inaccurate.
3. Believability / Reliability: IOA assesses believability—the extent to which independent observers arrive at consistent, repeatable measurement values. High IOA convinces researchers, clinicians, insurance payors, and parents that data variability reflects actual changes in the client's behavior rather than idiosyncratic observer differences.

Observer Drift

A primary reason for conducting ongoing IOA is detecting observer drift. Observer drift occurs when observers expand, contract, or unconsciously alter their operational definition of the target behavior over time, drifting away from the original clinical criteria established in the BIP. Without regular IOA, observer drift goes undetected, leading to false conclusions regarding client regression or progress.

---

Prerequisites for Valid IOA Data Collection

For an IOA assessment to be methodologically sound, five strict conditions must be fulfilled:

1. Identical Operational Definition: Both observers must evaluate the behavior using the exact same objective, technological definition.
2. Identical Measurement System: Both observers must utilize the exact same measurement system, interval lengths, and recording rules (e.g., both using 10-second Partial-Interval Recording).
3. Simultaneous Observation: Both observers must observe the exact same behavioral stream at the exact same time.
4. Completely Independent Observation: Observers must record data without communication, signaling, visual cuing, or looking at each other's data sheets. Synchronized vibrating timers or audio cues delivered through split headphones ensure interval synchronization without interpersonal signaling.
5. Observation of the Same Subject in the Same Context: Both observers must focus on the same learner throughout the observation window.

---

Continuous Measurement IOA Formulas & Step-by-Step Calculations

1. Total Count IOA

Total Count IOA is the simplest and crudest count IOA metric. It compares the overall total counts recorded by two observers across an entire session, without evaluating whether observers agreed on specific instances or intervals.

$\text{Total Count IOA} = \frac{\text{Smaller Count}}{\text{Larger Count}} \times 100$

Worked Clinical Example:

• Observer 1 (Technician) records 16 instances of property destruction.
• Observer 2 (Supervisor) records 20 instances of property destruction. $\text{Total Count IOA} = \frac{16}{20} \times 100 = 80.0\%$
• Clinical Critique: While fast and straightforward, Total Count IOA carries a severe flaw: it masks intra-session disagreements. Observer 1 could have recorded all 16 hits in the first 15 minutes, while Observer 2 recorded all 20 hits in the last 15 minutes. Despite scoring 80% agreement, they never agreed on a single actual occurrence!

2. Mean Count-per-Interval IOA

To resolve the limitation of Total Count IOA, Mean Count-per-Interval IOA divides the observation session into equal time intervals, calculates the Total Count agreement for each separate interval, and then averages these interval agreements across the entire session.

$\text{Mean Count-per-Interval IOA} = \frac{\sum_{i=1}^n \left(\frac{\text{Smaller Count}_i}{\text{Larger Count}_i}\right)}{n} \times 100$ (Note: Any interval where both observers record zero responses is scored as 1.00 or 100% agreement.)

Worked Clinical Example:

A 20-minute session is partitioned into four 5-minute intervals tracking vocal interruptions:

$\text{Mean Count-per-Interval IOA} = \frac{80.0\% + 100.0\% + 100.0\% + 33.3\%}{4} = \frac{313.3\%}{4} = 78.33\%$

• Clinical Interpretation: While Total Count IOA for this same data would be $\frac{7}{10} \times 100 = 70.0%$, Mean Count-per-Interval evaluates agreement within localized temporal windows, offering greater insight into observational consistency.

3. Exact Count-per-Interval IOA

Exact Count-per-Interval IOA is the most conservative and rigorous count agreement metric in Applied Behavior Analysis. It evaluates the percentage of total intervals in which both observers recorded the exact identical numerical count.

$\text{Exact Count-per-Interval IOA} = \frac{\text{Number of Intervals with Exact (100\%) Agreement}}{\text{Total Number of Intervals}} \times 100$

Worked Clinical Example (Using the same 4 intervals above):

• Interval 1 (4 vs. 5): Disagreement ($0%$)
• Interval 2 (2 vs. 2): Exact Agreement ($100%$)
• Interval 3 (0 vs. 0): Exact Agreement ($100%$)
• Interval 4 (1 vs. 3): Disagreement ($0%$)
• Exact Agreement occurred in exactly 2 out of 4 intervals. $\text{Exact Count-per-Interval IOA} = \frac{2}{4} \times 100 = 50.0\%$
• Clinical Critique: Exact Count-per-Interval sets an exceptionally high bar. Any discrepancy, even a difference of one count (e.g., 4 vs. 5), results in zero credit for that interval. If Exact Count-per-Interval IOA reaches $\ge 80%$, supervisors can have absolute confidence in data reliability.

4. Trial-by-Trial IOA (Discrete Trial Teaching)

Trial-by-Trial IOA is used for restricted operants, discrete trial teaching (DTT), or opportunity-based tasks where responses are scored as correct ($+$), incorrect ($-$), or prompted ($P$).

$\text{Trial-by-Trial IOA} = \frac{\text{Number of Agreed Trials}}{\text{Total Number of Trials Presented}} \times 100$

Worked Clinical Example:

Across 10 receptive identification trials, Technician A and Supervisor B independently record learner responses:

• Trials 1–7: Both record correct ($+$) $\implies 7$ agreements.
• Trial 8: Tech A records correct ($+$); Supervisor B records prompted ($P$) $\implies 1$ disagreement.
• Trial 9: Both record incorrect ($-$) $\implies 1$ agreement.
• Trial 10: Tech A records incorrect ($-$) ; Supervisor B records prompted ($P$) $\implies 1$ disagreement.
• Agreed Trials = $7 + 1 = 8$. Disagreed Trials = 2. $\text{Trial-by-Trial IOA} = \frac{8}{10} \times 100 = 80.0\%$

5. Total Duration IOA

Total Duration IOA compares the cumulative total duration of behavior recorded across an entire session by two independent observers.

$\text{Total Duration IOA} = \frac{\text{Shorter Duration}}{\text{Longer Duration}} \times 100$

Worked Clinical Example:

• Observer 1 records a cumulative total of 42 minutes of crying during a 3-hour session.
• Observer 2 records a cumulative total of 48 minutes of crying during the same session. $\text{Total Duration IOA} = \frac{42 \text{ minutes}}{48 \text{ minutes}} \times 100 = 87.5\%$

6. Mean Duration-per-Occurrence IOA

Mean Duration-per-Occurrence IOA is the most rigorous duration metric. It calculates the duration agreement for each discrete behavioral episode, sums the percentage agreements, and divides by the total number of episodes observed.

$\text{Mean Duration-per-Occurrence IOA} = \frac{\sum_{i=1}^n \left(\frac{\text{Shorter Duration}_i}{\text{Longer Duration}_i}\right)}{n} \times 100$

Worked Clinical Example:

Two observers record the duration of three discrete tantrum episodes:

• Episode 1: Observer A = 6 minutes; Observer B = 8 minutes $\implies \frac{6}{8} = 75.0%$
• Episode 2: Observer A = 12 minutes; Observer B = 12 minutes $\implies \frac{12}{12} = 100.0%$
• Episode 3: Observer A = 4 minutes; Observer B = 5 minutes $\implies \frac{4}{5} = 80.0%$ $\text{Mean Duration-per-Occurrence IOA} = \frac{75.0\% + 100.0\% + 80.0\%}{3} = \frac{255.0\%}{3} = 85.0\%$

---

Discontinuous / Interval IOA Formulas: The Inflation Traps

When evaluating time-sampling data (Partial, Whole, or MTS), selecting the correct interval IOA formula is critical. The wrong formula can artificially inflate agreement, disguising severe observer failure.

1. Interval-by-Interval (Total Interval) IOA

Interval-by-Interval IOA (also called Total Interval IOA) assesses agreement across all intervals, regardless of whether observers scored an occurrence or a non-occurrence.

$\text{Interval-by-Interval IOA} = \frac{\text{Agreed Intervals } (+/+ \text{ and } -/-)}{\text{Total Number of Intervals Observed}} \times 100$

The Fatal Flaw / Inflation Trap of Interval-by-Interval IOA:

Interval-by-Interval IOA is heavily influenced by the baseline prevalence of the behavior:

• For Low-Rate Behaviors: Observers easily achieve high agreement simply by agreeing that the behavior did not occur in most intervals.
• For High-Rate Behaviors: Observers easily achieve high agreement simply by agreeing that the behavior did occur in almost every interval.
• Consequently, Interval-by-Interval IOA is clinically invalid for extreme rate behaviors. Clinicians must use Scored-Interval or Unscored-Interval IOA instead.

2. Scored-Interval IOA (Occurrence IOA)

Scored-Interval IOA is ethically and methodologically mandated for LOW-RATE BEHAVIORS (behaviors occurring in $\le 30%$ of intervals).

• The Rule: The clinician completely discards intervals where BOTH observers recorded a non-occurrence ($-/-$). Agreement is calculated only across intervals in which at least one observer scored a positive occurrence.

$\text{Scored-Interval IOA} = \frac{\text{Agreed Occurrence Intervals } (+/+)}{\text{Agreed Occurrence } (+/+) + \text{Disagreed Intervals } (+/- \text{ or } -/+)} \times 100$

Comparative Proof (Low-Rate Behavior Vignette):

Two observers track low-rate severe aggression across 10 intervals. Aggression occurs rarely:

• Method A: Interval-by-Interval IOA:
  • Both agreed on Interval 1 ($+/+$), and both agreed on Intervals 2, 4, 5, 6, 7, 8, 9, 10 ($-/-$). That is 9 agreed intervals out of 10! $\text{Interval-by-Interval IOA} = \frac{9}{10} \times 100 = 90.0\% \quad (\text{Falsely indicating excellent reliability!})$
• Method B: Scored-Interval IOA (Mandated for Low-Rate Behaviors):
  • Discard the eight $(-/-)$ intervals (Intervals 2, 4, 5, 6, 7, 8, 9, 10).
  • Evaluate only intervals with at least one score: Interval 1 ($+/+$) and Interval 3 ($+/-$).
  • Agreed occurrences = 1. Disagreements = 1. $\text{Scored-Interval IOA} = \frac{1}{1 + 1} \times 100 = 50.0\% \quad (\text{Reveals true observational breakdown!})$
• Takeaway: Interval-by-Interval gave a misleading 90%, whereas Scored-Interval exposed that the technicians disagreed on 50% of actual aggressive episodes!

3. Unscored-Interval IOA (Non-Occurrence IOA)

Unscored-Interval IOA is ethically and methodologically mandated for HIGH-RATE BEHAVIORS (behaviors occurring in $\ge 70%$ of intervals).

• The Rule: The clinician completely discards intervals where BOTH observers recorded an occurrence ($+/+$). Agreement is calculated only across intervals in which at least one observer scored a non-occurrence.

$\text{Unscored-Interval IOA} = \frac{\text{Agreed Non-Occurrence Intervals } (-/-)}{\text{Agreed Non-Occurrence } (-/-) + \text{Disagreed Intervals } (+/- \text{ or } -/+)} \times 100$

Comparative Proof (High-Rate Behavior Vignette):

Two observers track continuous vocal stereotypy across 10 intervals. Stereotypy is pervasive:

• Method A: Interval-by-Interval IOA:
  • Both agreed on 8 occurrence intervals ($+/+$) and 1 non-occurrence interval ($-/-$). Total agreed = 9 out of 10. $\text{Interval-by-Interval IOA} = \frac{9}{10} \times 100 = 90.0\% \quad (\text{Falsely inflated by pervasive behavior!})$
• Method B: Unscored-Interval IOA (Mandated for High-Rate Behaviors):
  • Discard the eight $(+/+)$ intervals (Intervals 1 through 8).
  • Evaluate only intervals with at least one non-occurrence: Interval 9 ($+/-$) and Interval 10 ($-/-$).
  • Agreed non-occurrences = 1. Disagreements = 1. $\text{Unscored-Interval IOA} = \frac{1}{1 + 1} \times 100 = 50.0\%$
• Takeaway: Unscored-Interval IOA prevents pervasive occurrences from masking serious observer disagreement regarding when the behavior ceased.

---

Professional Standards for IOA Collection & Remediation

Under professional guidelines established by the Qualified Applied Behavior Analysis Credentialing Board (QABA) and the broader behavior analytic literature (e.g., Cooper, Heron, & Heward, 2020):

Frequency and Distribution of IOA Collection

1. Session Percentage: IOA must be collected in a minimum of 20% to 33% of sessions.
2. Phase Distribution: IOA must be distributed across all phases of intervention (baseline, treatment evaluation, maintenance, and generalization), not clustered conveniently in a single week.
3. Representative Sampling: IOA probes must be conducted across different times of day, different settings, and different behavior technicians to prevent systematic technician-specific bias.
4. Acceptable Benchmark: The professional clinical standard for acceptable IOA is $\ge 80%$ (in rigorous single-case research, the standard is often set at $\ge 90%$).

Systematic Supervisory Remediation Workflow for Low IOA ($< 80%$)

When IOA falls below 80%, the QASP-S must immediately enact a structured remediation protocol:

Step 1: Discrepancy Analysis ──► Step 2: Definition Audit ──► Step 3: Behavioral Skills Training ──► Step 4: Re-Calibration Probe
• Review interval sheets         • Check for ambiguities      • Instructions, Modeling,              • Collect IOA on video probe
• Identify disagreement patterns • Add non-examples/margins  • Rehearsal & Corrective Feedback     • Require >= 90% before resuming

1. Step 1: Discrepancy Analysis: The supervisor conducts an error analysis on the raw data sheets. Did the observers disagree on specific topographies (e.g., distinguishing a loud vocalization from an actual scream), or did they experience timing misalignments (e.g., one timer was 3 seconds ahead)?
2. Step 2: Operational Definition Audit: Ambiguous definitions are the primary cause of low IOA. The QASP-S refines the definition, adding objective boundary conditions, explicit onset/offset criteria, and concrete examples and non-examples.
3. Step 3: Behavioral Skills Training (BST): The supervisor retrains technicians using the four-step BST framework: (a) verbal and written instruction, (b) video modeling of target behaviors, (c) active rehearsal scoring calibrated video clips, and (d) immediate corrective feedback.
4. Step 4: Re-Calibration & Mastery Criterion: The technician must achieve at least 90% IOA across three consecutive video probes before resuming independent data collection with the client.

---

Comprehensive Summary Table of IOA Formulas

---

Interobserver Agreement Formula Selection Algorithm