6.1 Study Designs, Bias & Levels of Evidence in Clinical Research

Study Designs, Bias & Levels of Evidence in Clinical Research

Evidence-based medicine (EBM) is the conscientious, explicit, and judicious integration of the best available external clinical research evidence with individual clinical expertise and patient values. In contemporary Canadian pharmacy practice, evidence-based literature appraisal is not merely an academic exercise; it forms the foundation for evaluating formulary submissions, assessing drug efficacy claims, counseling patients, resolving complex drug therapy problems, and making autonomous therapeutic prescribing decisions within expanded scopes of practice.

---

The Hierarchy of Clinical Evidence in Pharmacotherapy

The evidentiary weight of clinical research is conventionally structured as a hierarchical pyramid. As one ascends the pyramid, study designs incorporate progressively greater methodological rigor to eliminate bias and confounding, yielding stronger inferences regarding causality and therapeutic efficacy.

                        ▲
                       / \
                      /   \
                     /     \
                    /  SR/  \
                   /  Meta-  \
                  /  Analyses \
                 /─────────────\
                /   Randomized  \
               / Controlled Trials\
              /───────────────────\
             /     Prospective     \
            /    Cohort Studies     \
           /─────────────────────────\
          /    Retrospective Cohort   \
         /      & Case-Control Studies \
        /───────────────────────────────\
       /      Cross-Sectional Studies    \
      /───────────────────────────────────\
     /      Case Series & Case Reports     \
    /───────────────────────────────────────\
   / Expert Opinion & Mechanistic Laboratory \
  /───────────────────────────────────────────\

1. Systematic Reviews and Meta-Analyses

At the apex of the evidence hierarchy, a systematic review synthesizes all available primary research addressing a tightly formulated clinical question using explicit, reproducible, and comprehensive literature search strategies. When primary studies are sufficiently homogenous in design, population, intervention, and outcome definitions, a meta-analysis quantitatively combines their numerical findings to generate a pooled statistical effect estimate (e.g., pooled Relative Risk or Odds Ratio). Meta-analyses substantially increase statistical power and precision, resolving conflicting results from individual underpowered trials.

2. Randomized Controlled Trials (RCTs)

Considered the gold standard primary experimental design for evaluating therapeutic efficacy. By randomly assigning participants to an experimental intervention or a comparative control (active comparator or placebo), RCTs distribute both known and unknown confounding variables equally across study arms at baseline, isolating the true pharmacological effect of the drug.

3. Cohort Studies

Observational studies where a defined group of exposed and unexposed individuals is followed over time to assess the incidence of specific health outcomes:

• Prospective Cohort Studies: Participants are enrolled prior to the occurrence of the clinical outcome and followed forward in chronological time (e.g., Framingham Heart Study, Nurses' Health Study). They permit direct calculation of incidence rates and relative risks ($RR$).
• Retrospective Cohort Studies: Both exposure and outcome have already occurred before the study initiates; investigators utilize existing health records or administrative claims databases (such as ICES in Ontario or RAMQ in Quebec) to reconstruct the cohort and track outcomes forward from a historical baseline.

4. Case-Control Studies

Observational, retrospective studies that begin with outcome identification. Investigators select individuals who have experienced the outcome (cases) and a comparable group without the outcome (controls), then look backward in time to determine the frequency or level of prior drug exposure. Case-control studies cannot directly calculate absolute incidence or relative risk; instead, they compute an Odds Ratio ($OR$). They are exceptionally resource-efficient for investigating rare diseases (e.g., rare adverse drug events such as Stevens-Johnson syndrome) or conditions with prolonged latency periods.

5. Cross-Sectional Studies

Studies that evaluate exposure and disease status simultaneously across a defined population at a single point in time (prevalence surveys). While valuable for determining public health disease prevalence and resource planning, cross-sectional designs cannot establish temporality (whether exposure preceded outcome), precluding causal inference.

6. Case Series and Individual Case Reports

Uncontrolled, descriptive accounts of single patients or small clusters of patients experiencing unusual clinical presentations, novel disease manifestations, or unexpected adverse drug reactions. They represent the lowest level of clinical evidence but serve as vital hypothesis-generating tools and early-warning signals in post-market pharmacovigilance.

---

Experimental vs. Observational Study Designs

The fundamental architectural distinction in clinical research lies between experimental and observational methodologies:

Experimental Architectures

In an experimental trial, the investigator actively assigns and manipulates the therapeutic exposure according to a predetermined protocol. Subtypes include:

• Parallel-Group RCT: Subjects are randomized to receive either Treatment A or Treatment B for the trial's duration.
• Crossover RCT: Each participant receives both treatments sequentially in a randomized order, serving as their own control. A critical requirement is an adequate washout period between treatment phases to eliminate residual pharmacological effects (carryover effect). Crossover designs are suitable only for stable, chronic conditions (e.g., mild hypertension, chronic stable angina) and cannot be used for curative interventions or acute events.
• Factorial Design: Evaluates two or more distinct interventions simultaneously in four combinations ($2 \times 2$ factorial), allowing researchers to assess both independent effects and potential pharmacological interactions.
• Cluster-Randomized Trials: Intact social or clinical units (e.g., entire community pharmacies, hospital wards, or primary care clinics) are randomized rather than individual patients, minimizing cross-contamination among subjects.

Observational Architectures

In observational research, investigators observe natural clinical practice without experimental intervention. While ethical and capable of assessing real-world populations over decades, observational studies cannot randomly balance patient characteristics. Consequently, observed associations between drug exposures and clinical outcomes may be distorted by underlying differences in patient health status.

---

Internal Validity vs. External Validity

When critically appraising clinical literature, pharmacists must evaluate both dimensions of study validity:

Internal Validity

Internal validity represents the degree to which the observed clinical results are free from systematic error (bias), confounding, and random error. It reflects whether the experimental intervention was truly responsible for the observed outcome difference within the study sample. Threats to internal validity include improper randomization, faulty blinding, differential loss to follow-up, and measurement inaccuracies.

External Validity (Generalizability)

External validity reflects the extent to which the trial's findings can be accurately generalized and applied to patients in routine clinical practice outside the study setting. A trial may exhibit flawless internal validity but possess severely compromised external validity if its inclusion criteria were excessively narrow.

---

Major Systematic Biases in Clinical Literature

Bias is any systematic (non-random) error in the design, conduct, analysis, or reporting of a study that results in a distorted estimate of an intervention's true effect.

                             TAXONOMY OF CLINICAL TRIAL BIASES
                                             │
         ┌───────────────────────────────────┼───────────────────────────────────┐
         ▼                                   ▼                                   ▼
   SELECTION BIAS                    INFORMATION BIAS                     ATTRITION BIAS
• Berkson's Bias               • Recall Bias (case-control)         • Unequal dropouts
• Healthy Worker Effect        • Detection / Surveillance Bias      • Selective withdrawal
• Volunteer / Self-Selection   • Performance Bias (unblinded)       • Protocol non-adherence

1. Selection Bias

Occurs when systematic differences exist between individuals selected into a study and those who are not, or in the manner participants are allocated to study arms:

• Berkson's Bias (Admission Rate Bias): Arises when hospitalized patients are utilized as controls in hospital-based case-control studies; hospitalized individuals have higher rates of comorbid conditions and medication exposures than the general population.
• Healthy Worker Effect: Occurs when an employed cohort is compared to the general population; employed individuals are systematically healthier than the general population, which includes disabled and chronically ill persons.
• Volunteer (Self-Selection) Bias: Individuals who actively volunteer for clinical trials tend to be more health-conscious, motivated, and adherent than non-volunteers.

2. Information (Measurement) Bias

Occurs when data regarding exposures or clinical outcomes are systematically mismeasured or misclassified:

• Recall Bias: A cardinal vulnerability of retrospective case-control studies. Patients who have suffered an adverse outcome (e.g., mothers of infants born with congenital anomalies) scrutinize their memories far more intensively and report past drug exposures more frequently than unaffected controls.
• Detection (Surveillance) Bias: Occurs when patients receiving an active treatment are monitored, examined, or tested more aggressively than control patients, resulting in higher diagnostic identification of mild or subclinical outcomes in the active arm.
• Performance Bias: Systematic differences in the care or co-interventions provided to participants outside the assigned intervention. Prevented by double-blinding (masking participants and healthcare providers) and double-dummy techniques when comparing drugs with different dosage forms or schedules.

3. Attrition Bias

Occurs when participants systematically drop out or are lost to follow-up at unequal rates or for different clinical reasons between study groups. For example, if patients in an experimental arm discontinue the trial because of severe adverse drug effects while control patients remain, an analysis that ignores dropouts will falsely portray the active drug as safer and more tolerable than it truly is.

---

Confounding and Methods of Control

The Confounding Triad

A confounder is an extraneous variable that distorts the apparent statistical association between an exposure and an outcome. To be a true confounder, a variable must satisfy three mandatory criteria simultaneously:

1. It must be independently associated with the exposure (without being caused by the exposure).
2. It must be an independent risk factor for the outcome (present even in the absence of the exposure).
3. It must not be an intermediate variable on the causal biological pathway between exposure and outcome.

                                   CONFOUNDER
                              (e.g., Cigarette Smoking)
                                   /          \
                         Associated            Associated
                        with Exposure          with Outcome
                                 /              \
                                ▼                ▼
                             EXPOSURE ───────► OUTCOME
                          (Coffee Intake)   (Pancreatic Cancer)

Methods to Control Confounding

Confounding can be controlled either during study design or during statistical analysis:

Design-Stage Control:

• Randomization: The only method capable of equally distributing both known and unmeasured/unknown confounders between study groups, provided the sample size is sufficiently large.
• Restriction: Limiting trial eligibility to a homogenous population lacking the confounder (e.g., enrolling only non-smokers). Limitation: severely limits external validity.
• Matching: Selecting controls who match cases on key confounding variables (e.g., matching each patient by exact age, biological sex, and baseline eGFR). Requires matched statistical analysis (e.g., conditional logistic regression).

Analysis-Stage Control:

• Stratification: Dividing the study cohort into distinct subgroups (strata) based on the confounding variable (e.g., analyzing smokers and non-smokers separately using Mantel-Haenszel methods). If the stratum-specific effect estimates are identical to each other but differ from the crude pooled estimate, confounding is confirmed.
• Multivariable Regression Modeling: Mathematical modeling that estimates the relationship between exposure and outcome while statistically holding other measured covariates constant:
  • Multiple Linear Regression: For continuous outcomes (e.g., changes in systolic blood pressure, reduction in LDL-C).
  • Multivariable Logistic Regression: For binary/dichotomous outcomes (e.g., stroke vs. no stroke; generates adjusted Odds Ratios).
  • Cox Proportional Hazards Regression: For time-to-event survival outcomes; generates adjusted Hazard Ratios ($HR$).
• Propensity Score Matching: A sophisticated technique in observational research where a logistic regression model calculates each patient's conditional probability (propensity score) of receiving a specific treatment given all observed baseline covariates. Patients who received the active drug are then matched to control patients with identical propensity scores, simulating randomized allocation for measured variables.

---

Analysis Paradigms: Intention-to-Treat vs. Per-Protocol vs. As-Treated

How clinical trialists handle non-compliant participants, dropouts, and treatment crossovers profoundly influences trial results:

┌────────────────────────────┬────────────────────────────┬────────────────────────────┐
│     INTENTION-TO-TREAT     │        PER-PROTOCOL        │         AS-TREATED         │
│           (ITT)            │            (PP)            │            (AT)            │
├────────────────────────────┼────────────────────────────┼────────────────────────────┤
│ "Once randomized, always   │ Evaluates only subjects    │ Categorizes participants   │
│ analyzed." Participants    │ who completed the study    │ based on the actual therapy│
│ analyzed in their assigned │ with full compliance and   │ they ingested, ignoring    │
│ group regardless of events.│ zero protocol violations.  │ original randomization.    │
├────────────────────────────┼────────────────────────────┼────────────────────────────┤
│ • Preserves baseline       │ • Evaluates maximum        │ • Completely destroys      │
│   randomization balance.   │   pharmacological efficacy │   randomization benefits.  │
│ • Reflects real-world      │   under ideal conditions.  │ • Reverts trial to an      │
│   clinical effectiveness.  │ • Prone to attrition bias. │   observational cohort.    │
│ • Gold standard for        │ • Mandatory secondary      │ • Severely confounded by   │
│   SUPERIORITY trials.      │   check in NON-INFERIORITY.│   indication.              │
└────────────────────────────┴────────────────────────────┴────────────────────────────┘

1. Intention-to-Treat (ITT) Analysis

Under the ITT principle, participants are evaluated within the randomized group to which they were initially assigned, regardless of whether they adhered to the protocol, experienced adverse effects, received the wrong medication, crossed over to the comparator arm, or dropped out prematurely.

• Why ITT is Essential: It strictly preserves the prognostic balance created by randomization, eliminates attrition bias, and provides an honest, pragmatic estimate of how a drug performs in routine practice where patients frequently miss doses or discontinue treatment.
• Limitation: In superiority trials, ITT tends to dilute differences between treatment arms (biasing toward the null hypothesis), making it a conservative standard. However, in non-inferiority trials, this dilution effect is hazardous: by driving treatment arms to appear similar, ITT can falsely make an inferior drug appear "non-inferior."

2. Per-Protocol (PP) Analysis

Evaluates only that subset of patients who completed the predetermined duration of treatment in strict compliance with the protocol without major deviations. While PP analysis answers the biological question—"Does the drug work when taken exactly as directed?"—it destroys the baseline comparability created by randomization, as compliant patients systematically differ from non-compliant patients in health literacy, disease severity, and lifestyle.