7.3 Exploratory Data Analysis in Clinical Populations

Exploratory Data Analysis in Clinical Populations

In healthcare data science and informatics, Exploratory Data Analysis (EDA) represents the critical investigative phase that precedes formal hypothesis testing, predictive machine learning modeling, or executive dashboard deployment. Pioneered by mathematician John W. Tukey in 1977, EDA is an analytical philosophy that emphasizes understanding data through graphical visualization, pattern discovery, anomaly detection, and assumption validation rather than relying prematurely on rigid probabilistic models.

For a Certified Health Data Analyst (CHDA), EDA is an indispensable clinical safeguard. Electronic Health Record (EHR) extracts, clinical trial registries, and billing data warehouses are inherently noisy, complex, and prone to systematic biases. Conducting rigorous EDA ensures that health data analysts uncover data quality flaws (e.g., impossible clinical values, unmapped codes), understand patient population heterogeneity, discover unexpected clinical relationships, and prevent severe analytical pitfalls such as Simpson's Paradox.

---

1. Core Objectives of Healthcare EDA

+---------------------------------------------------------------------------------------------------+
|                                 CORE OBJECTIVES OF HEALTHCARE EDA                                 |
+-----------------------------------+-----------------------------------+---------------------------+
| 1. POPULATION PROFILING           | 2. DATA QUALITY SCREENING         | 3. ASSUMPTION CHECKING    |
| - Characterize demographics,      | - Detect impossible values,       | - Test normality, linear- |
|   acuity, and case-mix index      |   missingness, duplicate records  |   ity, homoscedasticity   |
+-----------------------------------+-----------------------------------+---------------------------+
| 4. SUBGROUP / PHENOTYPE DISCOVERY | 5. ARTIFACT & BIAS IDENTIFICATION | 6. HYPOTHESIS GENERATION  |
| - Uncover high-utilizer clusters, | - Screen for zero-inflation,      | - Formulate grounded      |
|   rare disease sub-populations    |   digit preference, weekend dips  |   clinical inquiries      |
+-----------------------------------+-----------------------------------+---------------------------+

1. Population Profiling & Characterization: Establish baseline understanding of patient cohorts (age, biological sex, race/ethnicity, payer mix, primary diagnosis, comorbidity burdens).
2. Detecting Data Quality Flaws & Anomalies: Identify impossible clinical values (e.g., human heart rate of $0\text{ bpm}$ in an ambulatory patient, negative age, systolic blood pressure of $450\text{ mmHg}$), documentation errors, unmapped EHR flowsheets, and missingness mechanisms.
3. Validating Statistical Assumptions: Verify whether continuous variables meet the parametric assumptions (normality, linearity, homoscedasticity / equal variance) required for standard statistical models ($t$-tests, ANOVA, linear regression).
4. Identifying Clinical Subgroups & Phenotypes: Discover latent sub-populations (e.g., distinct diabetic patient phenotypes, sepsis sub-trajectories) through clustering and multimodal distribution analysis.
5. Detecting Systematic Artifacts & Operational Biases: Uncover institutional data collection biases, such as digit preference, zero-inflation, and weekend discharge patterns.
6. Hypothesis Formulation: Generate data-driven clinical and operational hypotheses to guide subsequent confirmatory research.

---

2. Univariate Exploratory Techniques

Univariate analysis examines the distribution, central tendency, dispersion, and outliers of a single variable in isolation.

+---------------------------------------------------------------------------------------------------+
|                                 UNIVARIATE EXPLORATION TOOLKIT                                    |
+-------------------+--------------------+--------------------+--------------------+----------------+
| FREQUENCY TABLES  | HISTOGRAMS         | DENSITY PLOTS (KDE)| BOX PLOTS (TUKEY)  | PARETO CHARTS  |
| - Categorical n, %| - Binned numeric   | - Smoothed continuous| - Median, IQR,   | - 80/20 Rule   |
| - Cumulative %    | - Skew, modalities |   distribution curve|   fences, outliers | - Readmissions |
+-------------------+--------------------+--------------------+--------------------+----------------+

1. Frequency Distributions & Cumulative Percentages

• Application: Essential for summarizing nominal and ordinal healthcare variables (e.g., patient discharge disposition, race/ethnicity, surgical wound classification).
• Structure: Displays absolute category counts ($n$), relative frequencies ($p_i = n_i / N$), and cumulative percentages ($\sum p_i$).

2. Histograms & Kernel Density Estimation (KDE)

• Application: Visualizes the probability distribution of continuous or discrete numerical variables by partitioning data into contiguous, non-overlapping bins along the horizontal axis, with vertical bar heights representing frequency counts.
• Bin Width Selection: The visual interpretation of a histogram is highly sensitive to bin width:
  • Sturges' Formula: $k = 1 + \log_2(n)$ (best for symmetric, moderate-sized datasets).
  • Freedman-Diaconis Rule: $\text{Bin Width} = 2 \times \frac{\text{IQR}}{\sqrt[3]{n}}$ (optimal for skewed healthcare data as it resists outlier distortion).
• Kernel Density Estimation (KDE): Constructs a smooth, non-parametric probability density curve over the histogram, making it easier to identify bimodal peaks, heavy tails, and distribution truncations.

3. Box-and-Whisker Plots (Tukey Boxplot)

Introduced by John Tukey, the boxplot provides a compact, 5-number visual summary that reveals center, spread, skewness, and outliers simultaneously.

+---------------------------------------------------------------------------------------------------+
|                                   ANATOMY OF A TUKEY BOXPLOT                                      |
+---------------------------------------------------------------------------------------------------+
                       * Extreme Outlier (> Q3 + 3.0*IQR)
                       o Mild Outlier (Q3 + 1.5*IQR to Q3 + 3.0*IQR)
                     ----- Upper Whisker: Min(Max_Value, Q3 + 1.5*IQR)
                       |   
                 +-----+-----+  <-- Q3 (75th Percentile / Upper Quartile)
                 |     |     |
                 |  ===+===  |  <-- Q2 (50th Percentile / MEDIAN LINE)
                 |     |     |
                 +-----+-----+  <-- Q1 (25th Percentile / Lower Quartile)
                       |   
                     ----- Lower Whisker: Max(Min_Value, Q1 - 1.5*IQR)
                       o Mild Outlier (< Q1 - 1.5*IQR)
+---------------------------------------------------------------------------------------------------+

• Box Boundaries: The lower edge represents $Q_1$ (25th percentile) and the upper edge represents $Q_3$ (75th percentile). The vertical span of the box equals the Interquartile Range (IQR), containing the middle 50% of patient data.
• Median Line: A horizontal line drawn inside the box at $Q_2$. If the median line is positioned closer to $Q_1$, the distribution is positively skewed; if closer to $Q_3$, it is negatively skewed.
• Whiskers: Extend from the box edges to the most extreme data points that fall within the inner fences: $\text{Upper Whisker} = \min(\max(x), Q_3 + 1.5\times\text{IQR})$ and $\text{Lower Whisker} = \max(\min(x), Q_1 - 1.5\times\text{IQR})$.
• Outliers: Individual points plotted beyond the whiskers. Values between $1.5\times\text{IQR}$ and $3.0\times\text{IQR}$ beyond the quartiles are mild outliers; values exceeding $3.0\times\text{IQR}$ are extreme outliers.

4. Pareto Charts & The 80/20 Rule

A Pareto chart is a dual-axis visualization that combines a descending bar chart (displaying individual category frequencies) with an ascending line graph (displaying cumulative percentage from 0% to 100%).

• The Pareto Principle (80/20 Rule): In healthcare quality management, approximately 80% of adverse outcomes, clinical errors, or financial costs stem from 20% of root causes or diagnostic categories.
• Healthcare Applications:
  • Hospital Readmissions Reduction: Identifying the vital 20% of primary discharge diagnoses (e.g., Congestive Heart Failure, COPD, Sepsis) that account for 80% of all 30-day hospital readmissions.
  • Clinical Documentation Improvement (CDI): Identifying the top physician query reasons that generate 80% of DRG assignment shifts.

---

3. Bivariate Exploratory Techniques

Bivariate analysis evaluates the statistical relationship, correlation, or dependency between two variables.

+---------------------------------------------------------------------------------------------------+
|                                 BIVARIATE ANALYSIS MATRIX                                         |
+-----------------------------------+-----------------------------------+---------------------------+
| VARIABLE COMBINATION              | EXPLORATORY VISUALIZATION         | STATISTICAL METRIC / TEST |
+-----------------------------------+-----------------------------------+---------------------------+
| Categorical vs. Categorical       | Cross-tabulation, Stacked Bar     | Chi-Square (χ²), Cramer's V
| Categorical vs. Continuous        | Grouped Boxplots, Violin Plots    | Mann-Whitney U, t-test    |
| Continuous vs. Continuous         | Scatter Plot, Loess Smooth Curve  | Pearson r, Spearman rho   |
+-----------------------------------+-----------------------------------+---------------------------+

1. Categorical vs. Categorical: Cross-Tabulations & Contingency Tables

• Structure: A two-dimensional grid displaying joint frequency distributions across two categorical variables (e.g., Smoking Status [Current, Former, Never] vs. COPD Exacerbation [Yes, No]).
• Analytics Metrics: Calculation of row percentages, column percentages, and total percentages. Statistical significance of association is evaluated via Pearson's Chi-Square ($\chi^2$) test of independence and strength of association via Cramér's $V$.

2. Categorical vs. Continuous: Grouped Boxplots & Violin Plots

• Grouped Boxplots: Multiple boxplots plotted side-by-side along a categorical axis. Allows immediate comparison of medians, spread (IQR), and outlier profiles across clinical departments or treatment groups (e.g., Inpatient Length of Stay stratified across Orthopedics, Cardiology, and General Surgery).
• Violin Plots: Combines a Tukey boxplot with a mirrored Kernel Density Estimation (KDE) plot on each side. Highly effective for detecting multi-modality within specific clinical subgroups that standard boxplots might obscure.

3. Continuous vs. Continuous: Scatter Plots & Correlation Analysis

• Scatter Plots: Plots individual patient observations as coordinate pairs $(x_i, y_i)$ on a Cartesian plane. Visualizes linear relationships, exponential curves, ceiling/floor boundary limits, heteroscedasticity (unequal variance across the range), and clustering.
• Choosing Correlation Metrics:
  • Pearson Product-Moment Correlation Coefficient ($r$):
    • Prerequisites: Measures the strength and direction of a linear relationship between two continuous, normally distributed interval/ratio variables without severe outliers.
    • Formula: $r = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum (x_i - \bar{x})^2 \sum (y_i - \bar{y})^2}}$, where $-1.0 \le r \le +1.0$.
  • Spearman Rank Correlation Coefficient ($\rho$):
    • Prerequisites: Non-parametric metric that evaluates the strength and direction of a monotonic relationship (whether linear or non-linear) between two continuous or ordinal variables. Robust against outliers and skewness.
    • Formula: $\rho = 1 - \frac{6 \sum d_i^2}{n(n^2 - 1)}$, where $d_i = \text{rank}(x_i) - \text{rank}(y_i)$.
    • Healthcare Use Case: Evaluating the relationship between patient age in years and ordinal Pain Severity Score (0–10).

---

4. Multivariate EDA & Clinical Stratification

Modern healthcare analytics problems involve complex interactions across dozens of clinical, demographic, and financial variables simultaneously.

+---------------------------------------------------------------------------------------------------+
|                                MULTIVARIATE EXPLORATORY METHODS                                   |
+-----------------------------------+-----------------------------------+---------------------------+
| CORRELATION HEATMAPS              | SCATTER PLOT MATRICES (PAIR PLOTS)| SUBGROUP STRATIFICATION   |
| - Color-encoded correlation matrix| - Grid of all pairwise scatter    | - Controlling for Simpson's
| - Rapid biomarker screening       |   plots and univariate densities  |   Paradox across cohorts  |
+-----------------------------------+-----------------------------------+---------------------------+

1. Correlation Heatmaps

• Mechanics: A color-encoded matrix displaying all pairwise Pearson ($r$) or Spearman ($\rho$) correlation coefficients across high-dimensional clinical datasets (e.g., vital signs, laboratory biomarkers, comorbidity counts).
• Analyst Utility: Immediately identifies multicollinearity (e.g., high correlation between BUN and Serum Creatinine, $r > 0.85$) prior to building multivariable regression models, preventing model instability.

2. Subgroup Stratification & Simpson's Paradox

A critical hazard in healthcare data analytics is Simpson's Paradox—a mathematical phenomenon where an apparent statistical trend, association, or treatment advantage observed in aggregate data disappears or reverses when the data is disaggregated into relevant clinical subgroups.

Healthcare Case Study: Hospital Surgical Mortality Comparison

Consider an aggregate analysis comparing 30-day post-surgical mortality rates between Hospital A (a regional academic medical center) and Hospital B (a community hospital):

Initial Aggregate Conclusion: At first glance, Hospital B appears to have superior surgical quality (lower mortality, 2.5% vs. 3.5%).

Now, conduct multivariate exploratory stratification by Patient Surgical Acuity / Case Complexity:

The True Clinical Reality: When stratified by patient acuity, Hospital A actually achieves lower mortality than Hospital B in high-acuity cases (4.0% vs. 5.0%) and comparable mortality in low-acuity cases. Hospital A's overall aggregate mortality appeared worse solely because 70% of its caseload consisted of high-acuity, high-risk patients, compared to only 10% at Hospital B.

• Analyst Takeaway: Health data analysts must always perform risk adjustment and subgroup stratification (by age, ASA score, comorbidity index, emergency status) before reporting clinical quality comparisons.

---

5. Detecting Clinical & Operational Data Artifacts

Healthcare datasets contain distinctive administrative and operational artifacts that can severely distort analytical models if not detected during EDA:

+---------------------------------------------------------------------------------------------------+
|                             HEALTHCARE DATA ARTIFACT TAXONOMY                                     |
+-----------------------------------+-----------------------------------+---------------------------+
| 1. ZERO-INFLATION                 | 2. DIGIT PREFERENCE / HEAPING     | 3. OPERATIONAL ARTIFACTS  |
| - Excess true zeros (no readmits) | - Rounding blood pressure to 0 / 5| - Weekend discharge drops |
| - Structural vs. sampling zeros   | - Weight rounded to nearest 5 kg  | - Shift-change note spikes|
+-----------------------------------+-----------------------------------+---------------------------+

1. Zero-Inflation

• Phenomenon: An excessive concentration of zero values that violates standard parametric probability distributions (e.g., standard Poisson or Gaussian curves).
• Healthcare Examples: Count of 30-day hospital readmissions (90%+ of patients have 0 readmissions), total coronary artery calcium scores, emergency department visits among healthy insured lives.
• Structural vs. Sampling Zeros:
  • Structural Zeros: Individuals who have zero probability of experiencing the event (e.g., prostate cancer diagnoses among biological females).
  • Sampling Zeros: Individuals who are at risk but did not experience the event during the observation window.
• Remediation: Apply Zero-Inflated Poisson (ZIP) or Zero-Inflated Negative Binomial (ZINB) regression models.

2. Digit Preference & Data Heaping

• Phenomenon: The systematic clustering of recorded numerical measurements around specific terminal digits (e.g., numbers ending in 0 or 5) caused by human rounding behavior during manual clinical data entry.
• Healthcare Examples:
  • Blood Pressure Measurements: Manual auscultatory sphygmomanometers exhibit massive digit preference for even numbers and multiples of 10 (e.g., 120/80, 130/85) compared to continuous automated oscillometric devices.
  • Patient Weight & Height: Clinicians frequently round patient weight to the nearest whole 5 or 10 kg/lbs during triage.
• Analytical Impact: Creates artificial spikes in histograms and can lead to misclassification around clinical treatment thresholds (e.g., classifying a patient as Stage 1 vs. Stage 2 hypertension based on a rounded $140\text{ mmHg}$ reading).

3. Operational & Temporal Artifacts

• Weekend Admission vs. Discharge Disparities: Inpatient hospital discharges drop significantly on Saturdays and Sundays due to reduced weekend staffing, physical therapy coverage, and post-acute SNF intake coordination. This creates artificial mid-week spikes in reported Length of Stay.
• Shift-Change Documentation Batching: Nursing flowsheets and vitals often show massive documentation clusters at 07:00 and 19:00 (nursing shift handoff) rather than smooth temporal logging throughout the shift.