4.2 Handling Missing Values, Outliers & Class Imbalance

Handling Missing Values, Outliers & Class Imbalance

Real-world machine learning datasets are rarely clean, complete, or balanced. Raw enterprise data frequently suffers from missing values, extreme sensor anomalies or recording errors (outliers), and severe class imbalance where the target event occurs in a minuscule fraction of total observations (e.g., credit card fraud, rare disease diagnosis, or hardware failure).

For the AWS Certified Machine Learning Engineer — Associate (MLA-C01) exam, you must know how to diagnose missingness mechanisms, select optimal imputation strategies, detect and winsorize outliers, and remediate severe class imbalance using both resampling techniques and cost-sensitive algorithmic adjustments while strictly avoiding data leakage.

+---------------------------------------------------------------------------------------------------+
|                         DATA CLEANSING & PREPROCESSING TAXONOMY                                   |
|                                                                                                   |
|   1. Missing Data Handling:                                                                       |
|      - Diagnosis: MCAR vs. MAR vs. MNAR                                                           |
|      - Remediation: Mean/Median/Mode, KNN, MICE, Missingness Indicators                           |
|                                                                                                   |
|   2. Outlier Detection & Treatment:                                                               |
|      - Detection: Z-Score (|Z| > 3), IQR Fences (1.5x IQR), Isolation Forest, RCF                 |
|      - Remediation: Winsorization (Clipping), RobustScaler, Imputation                            |
|                                                                                                   |
|   3. Class Imbalance Mitigation:                                                                  |
|      - Resampling (TRAIN SET ONLY): SMOTE, ADASYN, Random Under/Oversampling                      |
|      - Algorithmic: scale_pos_weight (XGBoost), Class Weights, Focal Loss                         |
|      - Evaluation: PR-AUC, F1-Score, Matthews Correlation Coefficient (MCC)                       |
+---------------------------------------------------------------------------------------------------+

---

1. Missing Data Mechanisms & Imputation Strategies

Before choosing an imputation technique, ML engineers must understand why data is missing. Statistical literature classifies missing data into three distinct mechanisms:

+---------------------------------------------------------------------------------------------------+
|                                 MISSINGNESS MECHANISMS COMPARISON                                 |
|                                                                                                   |
|   Mechanism   Definition                                              Optimal Strategy            |
|   ---------   -----------------------------------------------------   -------------------------   |
|   MCAR        Missingness is completely independent of observed and   Mean/Median/Mode or         |
|               unobserved data (e.g., dropped network packet).         Listwise deletion (if <5%)  |
|                                                                                                   |
|   MAR         Missingness depends systematically on other observed    Conditional Imputation      |
|               features, but not the missing value itself (e.g.,       (KNN, MICE, Regression)     |
|               younger users omit income, but age is recorded).                                    |
|                                                                                                   |
|   MNAR        Missingness depends directly on the value of the        Imputation + Missingness    |
|               missing variable itself (e.g., high-income earners      Indicator Flag              |
|               refuse to disclose income on surveys).                  (is_missing_income = 1)     |
+---------------------------------------------------------------------------------------------------+

Imputation Techniques

1. Univariate Imputation

• Mean Imputation: Replaces missing values with the arithmetic mean. Best suited for normally distributed numerical data with MCAR missingness. Caveat: Artificially reduces feature variance and distorts covariance with other variables.
• Median Imputation: Replaces missing values with the 50th percentile. Highly robust to skewed distributions and numerical outliers (e.g., income, house prices).
• Mode Imputation: Replaces missing values with the most frequent category. Standard for categorical features.
• Time Series Fills: In sequential or temporal data (e.g., stock prices, IoT sensor streams), use Forward Fill (ffill) (propagates the last valid observation forward) or Backward Fill (bfill); avoid mean/median imputation which destroys temporal autocorrelation.

2. Multivariate Imputation

• K-Nearest Neighbors (KNN) Imputation: Identifies the $k$ most similar records based on available features (using Euclidean or Gower distance) and computes a weighted average of their values. Captures complex non-linear relationships but is computationally expensive for large datasets.
• MICE (Multivariate Imputation by Chained Equations / IterativeImputer): Models each feature with missing values as a function of all other features through a series of iterative regression models.

3. The Missingness Indicator Pattern

When data is MNAR, the fact that a value is missing is itself a predictive signal. ML engineers should impute the missing feature (e.g., with median) AND generate a binary indicator column:

$$\text{is\_missing\_feature} = \begin{cases} 1 & \text{if feature was null} \\ 0 & \text{otherwise} \end{cases}$$

# Scikit-learn Pipeline with SimpleImputer and MissingIndicator
from sklearn.impute import SimpleImputer, MissingIndicator
from sklearn.pipeline import FeatureUnion
import numpy as np

# Combine median imputation with missing indicator flag
imputer_transformer = FeatureUnion(transformer_list=[
    ('median_imputer', SimpleImputer(strategy='median')),
    ('missing_flag', MissingIndicator(features='missing-only'))
])

---

2. Outlier Detection, Winsorization & Robust Scaling

Outliers are observations that deviate markedly from the overall distribution of the data. They can arise from measurement errors, corrupted sensor readings, data entry bugs, or genuine rare events.

+---------------------------------------------------------------------------------------------------+
|                                 OUTLIER DETECTION METHODOLOGIES                                   |
|                                                                                                   |
|   1. Z-Score (Parametric / Gaussian):                                                             |
|      Z = (X - mu) / sigma   ---> Flag if |Z| > 3.0 (captures outer 0.27% of Normal dist)          |
|                                                                                                   |
|   2. Interquartile Range (IQR / Non-Parametric / Tukey's Fences):                                 |
|      IQR = Q3 - Q1                                                                                |
|      Lower Fence = Q1 - 1.5 * IQR                                                                 |
|      Upper Fence = Q3 + 1.5 * IQR   ---> Flag if X < Lower Fence or X > Upper Fence               |
|      (Extreme outliers: use 3.0 * IQR)                                                            |
|                                                                                                   |
|   3. Algorithmic / Tree-Based:                                                                    |
|      - Isolation Forest: Isolates anomalies by random partitioning (anomalies have short paths). |
|      - SageMaker Random Cut Forest (RCF): Unsupervised anomaly scoring for streaming data.        |
+---------------------------------------------------------------------------------------------------+

Outlier Remediation Strategies

1. Winsorization (Clipping / Capping): Rather than deleting outlier rows (which reduces sample size and discards legitimate signal), extreme values are capped at predetermined percentiles (e.g., 1st and 99th percentiles) or at Tukey's IQR fences.

   # Winsorizing features using numpy clip
   q1 = train_df['transaction_amount'].quantile(0.25)
   q3 = train_df['transaction_amount'].quantile(0.75)
   iqr = q3 - q1
   lower_bound = q1 - 1.5 * iqr
   upper_bound = q3 + 1.5 * iqr
   
   train_df['amount_clipped'] = train_df['transaction_amount'].clip(lower_bound, upper_bound)
   

2. Robust Scaling (RobustScaler): Standardizes features by removing the median and scaling by the Interquartile Range ($X_{\text{scaled}} = \frac{X - Q_2}{Q_3 - Q_1}$), preventing outliers from compressing the normal feature range.
3. Logarithmic / Power Transforms: Applies $\log(1 + X)$ or Box-Cox / Yeo-Johnson transforms to compress heavily right-skewed fat-tailed distributions into approximate normal distributions.

---

3. Class Imbalance Mitigation Strategies

In domains such as cybersecurity intrusion detection, ad click prediction, and financial fraud, the positive (minority) class often represents less than 1% of total records. Standard loss functions will converge to predicting the majority class exclusively.

+---------------------------------------------------------------------------------------------------+
|                             CLASS IMBALANCE MITIGATION SPECTRUM                                   |
|                                                                                                   |
|   [DATA-LEVEL RESAMPLING]                      [ALGORITHMIC ADJUSTMENTS]                          |
|   (Applied to TRAIN set only)                  (Modify loss & gradients)                          |
|                                                                                                   |
|   - Random Undersampling (majority)            - scale_pos_weight (XGBoost / LightGBM)            |
|   - Random Oversampling (minority)             - Class-weighted loss (pos_weight / CrossEntropy)  |
|   - SMOTE (Synthetic Minority Over-sampling)   - Focal Loss (Down-weight easy examples)           |
|   - ADASYN (Adaptive Synthetic sampling)       - Decision threshold tuning (adjust ROC cutoff)    |
+---------------------------------------------------------------------------------------------------+

Data-Level Resampling Techniques

• Random Undersampling: Randomly discards majority class instances to equalize class ratios. Risk: Discards potentially vital information and reduces statistical power.
• Random Oversampling: Duplicates minority class instances randomly. Risk: Leads to severe overfitting as the model memorizes duplicated minority points.
• SMOTE (Synthetic Minority Over-sampling Technique): Synthesizes new minority instances by selecting a minority sample $\vec{x}_i$, finding its $k$-nearest minority neighbors, and generating a synthetic point along the line segment connecting them:

$$\vec{x}_{\text{new}} = \vec{x}_i + \lambda (\vec{x}_{\text{zi}} - \vec{x}_i) \quad \text{where } \lambda \in [0, 1]$$

• ADASYN (Adaptive Synthetic): An extension of SMOTE that creates more synthetic samples in regions where the minority class is surrounded by majority instances (i.e., harder-to-learn decision boundaries).
• Tomek Links / Edited Nearest Neighbors (ENN): Identifies and removes ambiguous or noisy majority class samples that lie directly adjacent to minority samples along the decision boundary, sharpening separation.

+---------------------------------------------------------------------------------------------------+
|                     CRITICAL EXAM RULE: THE RESAMPLING DATA LEAKAGE TRAP                          |
|                                                                                                   |
|   INCORRECT (DATA LEAKAGE):                                                                       |
|   [Raw Dataset] ---> [Apply SMOTE] ---> [Train / Test Split]  <=== CRITICAL FAILURE!              |
|   (Synthetic samples generated from test instances bleed directly into training set!)             |
|                                                                                                   |
|   CORRECT:                                                                                        |
|   [Raw Dataset] ---> [Train / Test Split] ---> [Apply SMOTE to TRAIN ONLY]                        |
|                                          ---> [Leave TEST / VALIDATION untouched]                 |
+---------------------------------------------------------------------------------------------------+

Algorithmic Approaches (No Resampling Required)

1. XGBoost / LightGBM scale_pos_weight

In gradient boosted trees, scale_pos_weight scales the gradient of the positive class relative to the negative class:

$$\text{scale\_pos\_weight} = \frac{\text{Number of Negative Instances}}{\text{Number of Positive Instances}}$$

import xgboost as xgb

num_neg = (y_train == 0).sum()
num_pos = (y_train == 1).sum()
scale_weight = num_neg / num_pos

# Configure XGBoost Estimator in SageMaker with scale_pos_weight
xgb_classifier = xgb.XGBClassifier(
    max_depth=6,
    learning_rate=0.1,
    n_estimators=300,
    scale_pos_weight=scale_weight,
    eval_metric='aucpr'  # Use PR-AUC as evaluation metric
)
xgb_classifier.fit(X_train, y_train)

2. Class Weights in Neural Networks & Scikit-Learn

In PyTorch or Scikit-learn, adjust the cross-entropy loss weights inversely proportional to class frequencies:

# Scikit-learn balanced class weighting
from sklearn.ensemble import RandomForestClassifier
clf = RandomForestClassifier(class_weight='balanced', random_state=42)

# PyTorch weighted BCE loss
import torch
import torch.nn as nn
pos_weight = torch.tensor([num_neg / num_pos])
criterion = nn.BCEWithLogitsLoss(pos_weight=pos_weight)

3. Focal Loss

Developed for dense object detection with extreme foreground-background class imbalance (1:1000), Focal Loss adds a modulating factor $(1 - p_t)^\gamma$ to standard cross-entropy to down-weight easy, well-classified examples and focus training on hard negative and minority instances:

$$\text{FL}(p_t) = -\alpha_t (1 - p_t)^\gamma \log(p_t)$$

Where $\gamma$ (focusing parameter, typically $\gamma = 2$) smoothly reduces the contribution of easy examples.

---

4. Evaluation Metrics for Imbalanced Datasets

Standard classification accuracy is completely invalid for imbalanced datasets. If a fraud dataset contains 99.9% legitimate transactions and 0.1% fraudulent transactions, a naive model that always predicts 'legitimate' achieves 99.9% accuracy while capturing zero fraud.

+---------------------------------------------------------------------------------------------------+
|                                 IMBALANCED EVALUATION METRICS MATRIX                              |
|                                                                                                   |
|   Metric              Formula                                      Best When                      |
|   -----------------   ------------------------------------------   ----------------------------   |
|   Precision           TP / (TP + FP)                               Cost of False Positive high    |
|   Recall (Sens.)      TP / (TP + FN)                               Cost of False Negative high    |
|   F1-Score            2 * (Precision * Recall) / (Prec + Rec)      Harmonic mean balance          |
|   PR-AUC              Area under Precision-Recall curve            Severe class imbalance (<5%)   |
|   ROC-AUC             Area under True Pos vs. False Pos curve      Moderate imbalance (>10%)      |
|   MCC                 (TP*TN - FP*FN) / sqrt((TP+FP)(TP+FN)...)    Balanced metric across all 4   |
+---------------------------------------------------------------------------------------------------+