5.2 Internal Validity, External Validity & Reliability in EBD

Internal Validity, External Validity & Reliability in EBD

Fundamental Principle: The credibility of any Evidence-Based Design claim hinges on three foundational scientific criteria: internal validity (did the design change actually cause the outcome?), external validity (will this design change work in our specific facility and population?), and reliability (was the outcome measured consistently and accurately?).

In the complex, high-stakes environment of healthcare capital projects, mistaking correlation for causation or blindly transplanting design concepts from one clinical context into an incompatible setting can lead to catastrophic operational failure. EDAC candidates must master the scientific anatomy of validity and reliability to critically appraise literature and design rigorous post-occupancy evaluations.

---

Deconstructing Variables: The Foundation of Validity

To evaluate validity, an EBD researcher must first clearly categorize the variables under investigation:

• Independent Variable (IV): The environmental design condition manipulated or measured (e.g., single-patient rooms vs. double-occupancy rooms; decentralized vs. centralized nurse stations; circadian tunable LED lighting vs. static fluorescent lighting; sound-absorbing ceiling tiles with NRC 0.90 vs. hard gypsum plaster).
• Dependent Variable (DV): The measurable clinical, operational, financial, or behavioral outcome expected to change in response to the design intervention (e.g., hospital-acquired Clostridioides difficile infection rates; patient fall incidence per 1,000 patient-days; nurse walking distance in kilometers per 12-hour shift; average post-surgical analgesic consumption in morphine milligram equivalents [MME]).
• Extraneous / Confounding Variables: Uncontrolled factors that covary with the independent variable and obscure the true causal relationship (e.g., changes in clinical staffing ratios, introduction of new electronic health records, seasonal influenza surges, or new physician leadership).

---

Measurement Validity and Reliability: The EDAC Definitions

EDAC Study Guide 2 defines the two terms at the level of measurement tools:

• Validity is the extent to which a measurement tool measures what it is intended to measure.
• Reliability is the degree to which a measurement tool produces consistent or similar results on the same phenomenon at different times or when used by different people.

The same ideas scale up to whole studies: internal validity asks whether the study design supports a causal conclusion, and external validity asks whether the findings transfer to other settings.

---

Internal Validity: Demonstrating True Causality

Internal validity reflects the degree of certainty that the independent environmental variable directly caused the observed change in the dependent clinical outcome, rather than extraneous, unmeasured factors.

In laboratory science, researchers achieve high internal validity by isolating subjects in sterile, sealed chambers. In operational hospitals, patients and clinicians operate within open, dynamic socio-technical systems. Built-environment researchers must identify and mitigate the classic threats to internal validity described by Campbell and Stanley:

Classic Threats to Internal Validity in EBD

1. History:

   • Definition: Specific external events occurring between pretest and posttest measurements that are concurrent with, but unrelated to, the architectural intervention.
   • Healthcare Example: A hospital renovates an orthopedic surgery floor to install decentralized charting stations, hoping to reduce nurse response times and patient falls. Concurrently, the hospital implements a mandatory hospital-wide "Call Don't Fall" hourly purposeful rounding campaign. If falls decrease by 30%, the drop may be caused by the rounding protocol (history) rather than the physical station configuration.

2. Maturation:

   • Definition: Biological, physiological, or psychological processes occurring naturally within human subjects over time, independent of environmental conditions.
   • Healthcare Example: In a study tracking post-surgical pain in rooms with nature views over a 7-day inpatient stay, patients naturally experience decreasing pain and reduced narcotic requirements as surgical incisions heal (biological maturation). A valid study must compare healing curves against a matched control group rather than assuming pain reduction resulted solely from window views.

3. Testing (Pretest Sensitization):

   • Definition: The process of administering a pretest measurement alters the participants' subsequent behavior or survey responses during posttest measurements.
   • Healthcare Example: Surveying nurses about noise levels in a baseline study raises their awareness of acoustic disruptions. In the post-occupancy evaluation, nurses may report heightened annoyance simply because they were sensitized to listen for noise, even if decibel levels remained identical.

4. Instrumentation (Instrumentation Drift):

   • Definition: Changes in the calibration of measuring instruments, diagnostic criteria, or human observer rating standards between pretest and posttest.
   • Healthcare Example: A hospital evaluates the impact of copper-alloy touch surfaces on hospital-acquired infections (HAIs). Midway through the study, the hospital shifts from manual microbiological swab culturing to high-sensitivity polymerase chain reaction (PCR) genetic assays, or the CDC updates NHSN diagnostic definitions for catheter-associated urinary tract infections (CAUTIs). The apparent change in infection rates reflects altered instrumentation, not surface performance.

5. Statistical Regression (Regression to the Mean):

   • Definition: The statistical phenomenon where units or subjects selected on the basis of extreme, outlier baseline scores naturally drift back toward their historical average during subsequent measurements.
   • Healthcare Example: Hospital leadership selects Medical Unit 3 for an emergency $2 million acoustic and lighting renovation because its fall rate spiked to an all-time high of 14 falls per 1,000 patient-days in Q3. In Q4, following the renovation, the fall rate drops to 6 falls per 1,000 patient-days. Some or much of this decline may be regression to the mean: an extreme spike often moves back toward the usual level even without any renovation.

6. Selection Bias:

   • Definition: Systematic differences in the characteristics of subjects assigned to the intervention group compared to the control group at baseline.
   • Healthcare Example: A hospital opens a newly constructed inpatient tower with private rooms and retains its older wing with semi-private rooms. Nurse managers triage healthier, younger, ambulatory patients into the new wing while placing older, frail, high-acuity geriatric patients in the older wing. Comparing fall rates between the two wings produces an illusion of architectural superiority driven entirely by selection bias.

7. Experimental Mortality (Attrition):

   • Definition: The differential loss of participants or clinical staff from study groups over time, skewing the remaining sample.
   • Healthcare Example: Following the transition from a centralized nurse station to a decentralized alcove model, several experienced senior nurses who dislike working in physical isolation transfer to another hospital. A post-occupancy survey of the remaining staff reveals glowing satisfaction scores—because the dissatisfied staff left (attrition bias).

---

External Validity: The Challenge of Generalizability

External validity (generalizability) represents the extent to which research findings from a specific study setting and patient sample can be reliably applied to other healthcare organizations, clinical specialties, geographic regions, or patient demographics.

A study may have strong internal validity (good confidence that an intervention worked in Facility X) while having little external validity for Facility Y.

Contextual Boundaries and Generalizability Traps

Primary Threats to External Validity

• Interaction of Setting and Intervention: Environmental interventions interact uniquely with the physical plant (e.g., an acoustic ceiling tile that performs exceptionally in a dry, low-humidity climate may harbor fungal spores and fail infection-control standards in a humid coastal hospital).
• Population Specificity: Patient age, cognitive status, mobility, and cultural background moderate environmental responses (e.g., abstract or ambiguous art that alert younger patients tolerate may be misread and distressing for patients with dementia or delirium).
• Novelty and Hawthorne Effects: When a brand-new, $500-million architectural masterpiece opens, staff morale, donor enthusiasm, and patient satisfaction temporarily soar simply because of the "new car smell" and excitement of novelty. Studies evaluated very soon after opening can show inflated results that fade as the novelty wears off.

---

Measurement Reliability: Precision and Stability

Reliability refers to the consistency, dependability, and repeatability of a measurement tool or observational protocol. If an instrument measures the exact same physical or behavioral phenomenon under identical conditions, it must yield identical results.

The Three Types of Reliability in Built-Environment Research

1. Test-Retest Reliability:

   • Concept: Stability of an instrument over repeated administrations across time.
   • Application: Deploying a Class 1 sound-level meter to measure background ambient equivalent sound pressure (LAeq) in an unoccupied operating suite on Monday at 2:00 AM and again on Wednesday at 2:00 AM. If readings fluctuate wildly (e.g., 38 dBA vs. 58 dBA) under identical HVAC operating loads, the instrument lacks test-retest reliability.

2. Inter-Rater Reliability (Inter-Observer Agreement):

   • Concept: The degree of consensus between two or more independent researchers observing and coding the exact same physical behavior or environmental condition simultaneously.
   • Application: In behavioral mapping studies, researchers observe nurse locations, hand-hygiene events, and patient-family interactions. If Observer A records 45 handwashing events while Observer B records 18 handwashing events during the same observation shift, the protocol is unreliable.
   • Statistical Standard: Cohen's kappa (κ) is a commonly used statistic for inter-rater agreement on categorical data; it adjusts for agreement expected by chance.

$\kappa = (P_o - P_e) / (1 - P_e)$

Where P_o is the observed proportional agreement and P_e is the expected hypothetical agreement by chance.

(Interpretation bands vary by author; this table follows a commonly cited healthcare research scale.)

1. Internal Consistency Reliability:
   • Concept: The degree to which different survey items measuring the same underlying psychological construct produce similar scores.
   • Application: On a post-occupancy nurse survey measuring "Perceived Acoustic Distraction," researchers use four separate Likert-scale questions. Cronbach's Alpha (α) quantifies internal consistency. A scale must achieve α ≥ 0.70 (preferably ≥ 0.80) to be deemed reliable.

---

The Interplay Between Reliability and Validity

A critical conceptual distinction is the relationship between validity and reliability:

   UNRELIABLE & INVALID           RELIABLE BUT INVALID           RELIABLE & VALID
   (Scattered off-target)       (Tightly clustered off-target)  (Tightly clustered bullseye)
        ○   ○                         ● ●
          ○                             ● ●                        ●●●
      ○       ○                                                    ●●●
        ○   ○

• Reliability is a necessary, but insufficient, condition for validity.
• A measurement instrument can be perfectly reliable while being completely invalid (e.g., an uncalibrated digital scale that consistently reads exactly 5.0 pounds too heavy is 100% reliable, but completely invalid for measuring true weight).
• Conversely, a measurement instrument cannot be valid if it is unreliable. If an acoustic meter produces random, erratic readings, it cannot possibly reflect the true acoustic performance of the architectural ceiling.

[!TIP]

EXAM TIP: Diagnostic Checklist for Validity vs. Reliability

When tackling scenario questions, use this rapid classification:

• Internal Validity Question: "Did the architectural intervention cause this clinical change, or was it an operational confound (history, maturation, regression)?"
• External Validity Question: "Can we take this research finding from an adult suburban hospital and apply it to an urban pediatric or behavioral health unit?"
• Reliability Question: "Did two independent observers agree on their behavioral counts (for example, a high Cohen's kappa), or did the tool produce consistent readings over time?"