Validity, Reliability, IOA, Procedural Integrity, and Dosage
Key Takeaways
- Validity asks whether the data measure the intended behavior and dimension.
- Reliability concerns consistency of measurement; IOA estimates agreement between observers.
- High IOA does not prove data are valid if observers share the same flawed definition.
- Procedural integrity and dosage data help interpret whether implementation matched the plan.
Data Quality and Implementation Data
Validity asks whether measurement captures the intended behavior and dimension. Reliability asks whether measurement is consistent. IOA is a formal check of observer agreement, but it is not proof that observers measured the right thing.
Quality Decision Chain
| Problem pattern | First check |
|---|---|
| Observers disagree | Definition, training, scoring rules, IOA method |
| Data do not match the clinical question | Validity of the measure and dimension |
| Intervention data are flat | Procedural integrity and dosage before rejecting the plan |
| Change appears only on some days | Setting events, schedule, observer coverage, representativeness |
Procedural integrity measures whether the plan was implemented as written. Dosage measures the amount of exposure, such as minutes, sessions, opportunities, or trials.
If outcome data show no improvement but integrity is low, the most defensible decision is usually to improve implementation and collect more data. If integrity and dosage are adequate, then modification may be warranted.
Exam Tie-In
Do not let IOA distract from validity. Observers can agree perfectly on the wrong response class or wrong dimension. On scenario items, check whether the data answer the referral question, whether the plan was implemented, and whether the client had enough exposure before changing treatment.
Two observers have 96% IOA for disruption, but the definition includes crying, refusal, aggression, and leaving the area as one broad category. The team cannot tell which response changed. What is the main concern?
A graph shows no improvement after an intervention begins. Procedural integrity data show staff implemented only 52% of required steps. What should the BCBA do first?
A BCBA tracks the number of teaching opportunities delivered per session in addition to percent correct. What implementation variable is being measured?