2.3 Comparing Two Categorical or Two Continuous Features

2.3 Comparing Two Categorical or Two Continuous Features

Comparing two features means quantifying whether knowing one tells you anything about the other. The right statistic is determined entirely by the pair of types involved, and picking the wrong one is the mistake the exam is testing.

Two Continuous Features

Pearson's $r$ measures the strength and direction of a linear relationship on $[-1, 1]$. Spearman's $\rho$ is Pearson's correlation computed on the ranks, so it detects any monotonic relationship and is robust to outliers and to non-linear but order-preserving transforms.

from pyspark.ml.feature import VectorAssembler
from pyspark.ml.stat import Correlation

# Single pair
print(df.stat.corr("monthly_charges", "total_charges"))                 # Pearson
print(df.stat.corr("monthly_charges", "total_charges", method="spearman"))

# Full matrix over several numeric columns
numeric = ["tenure_months", "monthly_charges", "total_charges", "support_tickets"]
vec_df = VectorAssembler(inputCols=numeric, outputCol="features").transform(df)
pearson_matrix = Correlation.corr(vec_df, "features", "pearson").head()[0]
spearman_matrix = Correlation.corr(vec_df, "features", "spearman").head()[0]
print(pearson_matrix.toArray())

Choosing between them

The trap: $r \approx 0$ does not mean "no relationship". A perfectly symmetric U-shaped relationship has a Pearson correlation of approximately zero while being completely deterministic. Confirm with a scatter plot before dropping a feature.

Two Categorical Features

Start with the contingency table — the joint counts of every level pair:

crosstab = df.stat.crosstab("contract_type", "payment_method")
display(crosstab)

Then test whether the two are independent. The chi-square test of independence compares observed cell counts against the counts expected if the features were unrelated:

$\chi^2 = \sum_{i,j} \frac{(O_{ij} - E_{ij})^2}{E_{ij}}, \qquad E_{ij} = \frac{R_i C_j}{N}$

A small p-value means the features are not independent — knowing one shifts the distribution of the other.

from pyspark.ml.feature import StringIndexer, VectorAssembler
from pyspark.ml.stat import ChiSquareTest

indexed = StringIndexer(inputCols=["contract_type", "payment_method"],
                        outputCols=["contract_idx", "payment_idx"]).fit(df).transform(df)
vec = VectorAssembler(inputCols=["contract_idx"], outputCol="features").transform(indexed)

result = ChiSquareTest.test(vec, "features", "payment_idx").head()
print("p-values:", result.pValues)
print("degrees of freedom:", result.degreesOfFreedom)
print("statistics:", result.statistics)

Statistical significance is not effect size

On a 50-million-row table, almost every chi-square test returns a p-value near zero, because the test's power grows with $N$. Report an effect size alongside it — Cramér's V rescales the statistic to $[0, 1]$:

$V = \sqrt{\frac{\chi^2}{N \cdot \min(r - 1,\; c - 1)}}$

where $r$ and $c$ are the number of levels in each feature. Values near 0 mean the association is negligible regardless of how small the p-value is.

Cell-count requirement

The chi-square approximation degrades when expected cell counts are very small (the conventional guidance is at least 5 per cell). With many rare levels, collapse the tail into an "Other" bucket before testing.

---

Detecting Multi-Collinearity & Feature Interactions

Multi-collinearity occurs when two or more independent predictor features are highly linearly correlated. While tree-based ensembles (such as Random Forests and XGBoost) are relatively resilient to collinearity during inference, linear and logistic regression models suffer from inflated standard errors, unstable coefficient weights, and degraded interpretability.

Computing Pearson Correlation in PySpark

To compute the pairwise Pearson correlation coefficient between two numeric columns in PySpark without collecting the entire dataset to the driver:

# Distributed pairwise Pearson correlation calculation
corr_charges = df_features.stat.corr("monthly_charges", "total_charges")
print(f"Pearson Correlation: {corr_charges:.4f}")

# Cross-tabulation for categorical feature interaction
crosstab_df = df_features.stat.crosstab("contract_type", "churn_label")
display(crosstab_df)

Full Correlation Matrix via pandas API on Spark

When exploring dozens of features, computing a full correlation matrix is streamlined using the pyspark.pandas (formerly Koalas) API, which scales across Spark worker nodes:

import pyspark.pandas as ps

# Convert Spark DataFrame to distributed pandas-on-Spark DataFrame
ps_df = df_features.select("tenure_months", "monthly_charges", "total_charges").pandas_api()

# Compute distributed correlation matrix
corr_matrix = ps_df.corr(method="pearson").to_pandas()
print(corr_matrix)

---

Target Distribution Analysis: Imbalance & Skewness

Diagnosing the distribution of the target variable is one of the most critical steps in the EDA phase, as it dictates the entire downstream modeling strategy.

Classification Target Imbalance

In classification tasks (e.g., fraud detection, anomaly detection, churn prediction), extreme class imbalance (such as 99% majority vs. 1% minority) causes standard accuracy metrics to become deceptive. An algorithm predicting the majority class exclusively would achieve 99% accuracy while exhibiting 0% recall on the minority target.

During EDA, if class proportions diverge significantly from a 50/50 or 60/40 ratio, the data scientist must plan for:

• Utilizing Precision-Recall AUC (PR-AUC), F1-Score, or Balanced Accuracy rather than ROC-AUC or standard classification accuracy.
• Applying class-weight balancing (e.g., scale_pos_weight in XGBoost/LightGBM or weightCol in Spark ML).
• Evaluating oversampling (SMOTE) or stratified sampling techniques during train/validation splitting.

Regression Target Skewness

In regression tasks (e.g., home price prediction, lifetime customer value, transaction volume), target distributions are frequently right-skewed with long positive tails. Severe skewness violates the normality assumptions of linear regression residuals and causes gradient descent updates to be dominated by extreme values.

EDA visual inspection via histograms and box plots reveals whether power transformations (such as $\ln(y+1)$) should be applied prior to model fitting.

---

Pre-Modeling Data Quality Assessment Checklist