2.4 Removing Outliers with Standard Deviation and IQR

2.4 Removing Outliers with Standard Deviation and IQR

In machine learning feature engineering, an outlier is an observation that deviates substantially from the overall distribution of the dataset. Outliers can arise from measurement errors, sensor failures, data entry typos, data corruption, or genuine rare real-world phenomena (e.g., high-net-worth individuals in wealth management or flash-crash volume spikes in financial trading).

Understanding the mathematical nature of outliers and their downstream impact on specific algorithms is a core competency tested on the Databricks Certified Machine Learning Associate exam.

---

Algorithm Sensitivity to Outliers

Machine learning algorithms exhibit vastly different degrees of sensitivity to extreme values. The decision to invest time in outlier detection and remediation depends entirely on the chosen model family:

---

Mathematical Outlier Detection Methods

The Interquartile Range (IQR) Rule (Tukey's Fences)

The IQR rule is a robust, non-parametric method that makes no assumptions regarding the underlying probability distribution of the data. It is resistant to extreme skewness.

$\text{IQR} = Q_3 - Q_1$ $\text{Lower Inner Fence} = Q_1 - 1.5 \times \text{IQR}$ $\text{Upper Inner Fence} = Q_3 + 1.5 \times \text{IQR}$

Observations falling outside $[\text{Lower Inner Fence}, \text{Upper Inner Fence}]$ are classified as mild outliers. Observations beyond $3.0 \times \text{IQR}$ are classified as extreme outliers.

# Calculating IQR Fences using PySpark percentile_approx
from pyspark.sql import functions as F

quantiles = df.select(
    F.percentile_approx("annual_income", [0.25, 0.75], 10000).alias("q")
).collect()[0]["q"]

q1, q3 = quantiles[0], quantiles[1]
iqr = q3 - q1
lower_fence = q1 - 1.5 * iqr
upper_fence = q3 + 1.5 * iqr

print(f"Q1: {q1}, Q3: {q3}, IQR: {iqr}")
print(f"Valid Range: [{lower_fence}, {upper_fence}]")

# Flag outliers in PySpark
df_flagged = df.withColumn(
    "is_income_outlier",
    (F.col("annual_income") < lower_fence) | (F.col("annual_income") > upper_fence)
)

The Z-Score Method (Standard Deviation Rule)

The Z-score measures how many standard deviations ($\sigma$) an observation $x$ lies away from the arithmetic mean ($\mu$):

$z = \frac{x - \mu}{\sigma}$

Under a standard Gaussian (normal) distribution:

• $68.27%$ of values lie within $|z| \le 1$
• $95.45%$ of values lie within $|z| \le 2$
• $99.73%$ of values lie within $|z| \le 3$

Threshold: An observation is flagged as an outlier if $|z| > 3.0$.

Exam Warning: The Z-score assumes the underlying data is normally distributed. Furthermore, the calculation of $\mu$ and $\sigma$ is itself distorted by extreme outliers (the masking effect). If data is non-normal or skewed, use the IQR method or the Median Absolute Deviation (MAD) rather than standard Z-scores.

$\text{MAD} = \text{median}(|x_i - \text{median}(x)|)$ $\text{Modified } Z_i = \frac{0.6745 \times (x_i - \text{median}(x))}{\text{MAD}}$

---

Choosing Between the Two Rules

On a right-skewed feature such as income, the Z-score rule flags almost nothing on the left and far too little on the right, because a handful of very large values has already pushed the mean up and the standard deviation out. The IQR rule is the safer default and is what Databricks material generally demonstrates.

Filtering in Spark, end to end

from pyspark.sql import functions as F

q1, q3 = df.approxQuantile("annual_income", [0.25, 0.75], 0.01)
iqr = q3 - q1
lower, upper = q1 - 1.5 * iqr, q3 + 1.5 * iqr

clean_df = df.filter((F.col("annual_income") >= lower) & (F.col("annual_income") <= upper))

# Standard-deviation alternative, computed in one distributed pass
stats = df.select(F.mean("annual_income").alias("mu"),
                  F.stddev("annual_income").alias("sigma")).first()
z_clean_df = df.filter(F.abs((F.col("annual_income") - stats["mu"]) / stats["sigma"]) <= 3)

Two implementation rules the exam rewards:

1. Compute the thresholds on the training split only. Fences derived from the full dataset leak test-set distribution information into training, exactly as an imputer fitted on everything would.
2. Never collect() the column. approxQuantile and the aggregate functions run distributed; pulling millions of values to the driver to sort them is the wrong answer even when it is written in valid Python.

---

Outlier Remediation Strategies

Once outliers are detected, data scientists select an appropriate treatment strategy:

1. Trimming / Filtering (Row Deletion):
   • Approach: Drop rows containing outlier values: df.filter((col("x") >= lower) & (col("x") <= upper)).
   • When to use: Obvious data corruption or recording errors where the observation is invalid.
   • Risks: Can remove genuine rare events, reduce sample size, and introduce selection bias.
2. Winsorization (Percentile Capping):
   • Approach: Cap extreme values at predetermined lower and upper percentiles (e.g., 1st and 99th percentiles, or 5th and 95th percentiles).
   • When to use: When the observation is valid and the row must be preserved, but extreme magnitudes would destabilize parametric models.
   • PySpark implementation:

     # Winsorize / Cap feature values at 1st and 99th percentiles
     p01, p99 = df.select(
         F.percentile_approx("transaction_amount", [0.01, 0.99], 10000).alias("p")
     ).collect()[0]["p"]
     
     df_capped = df.withColumn(
         "transaction_amount_capped",
         F.when(F.col("transaction_amount") < p01, p01)
          .when(F.col("transaction_amount") > p99, p99)
          .otherwise(F.col("transaction_amount"))
     )
     

3. Outlier Indicator Feature Encoding:
   • Approach: Create a binary indicator column ($1$ if outlier, $0$ otherwise) before capping or imputing.
   • Benefit: Preserves the informational signal that an anomaly occurred while preventing numerical instability in the primary feature column.