2.7 One-Hot Encoding and When It Is or Is Not Appropriate

2.7 One-Hot Encoding and When It Is or Is Not Appropriate

Raw enterprise data in the Lakehouse is rarely formatted as pure numerical vectors ready for estimator ingestion. Categorical variables (such as zip codes, device types, education levels, and transaction statuses) must be mathematically transformed into numeric representations, and continuous numeric features of varying scales must be standardized or normalized.

Mastering feature transformation pipelines—and understanding how they behave in distributed Spark ML vs. single-node scikit-learn environments—is a critical domain on the Databricks Machine Learning Associate exam.

---

Categorical Data Taxonomy: Nominal vs. Ordinal

Before selecting an encoding strategy, the data scientist must identify whether categorical features possess an inherent mathematical ordering:

                           CATEGORICAL DATA TYPES
                                     │
        ┌────────────────────────────┴────────────────────────────┐
        ▼                                                         ▼
     NOMINAL                                                   ORDINAL
(No Natural Order)                                        (Explicit Hierarchy)
• State: [CA, NY, TX]                                     • Education: [High School < BS < MS < PhD]
• Device: [iOS, Android, Web]                             • Customer Tier: [Bronze < Silver < Gold]
• Encoding: One-Hot Encoding,                             • Encoding: Ordinal Integer Mapping
  Target Encoding, Frequency Encoding                       (e.g., Bronze=1, Silver=2, Gold=3)

Exam Trap: Applying simple integer/label encoding (e.g., Red=1, Green=2, Blue=3) to nominal variables imposes an artificial linear hierarchy. A linear regression model would incorrectly assume that Blue is three times the value of Red and that Green is the exact mathematical midpoint. Use One-Hot Encoding for nominal data.

---

Categorical Encoding Techniques & Tradeoffs

---

When One-Hot Encoding Is the Wrong Tool

The exam asks explicitly about the model types and datasets where one-hot encoding is inappropriate. There are four recognised cases.

High-cardinality features

A zip_code column with 40,000 levels becomes 40,000 sparse columns. Memory and training time grow with the level count, most columns are almost always zero, and each individual indicator carries too little signal to be selected by a model. Prefer target encoding with out-of-fold smoothing, frequency encoding, hashing, or a top-K plus "Other" grouping.

Ordinal features

Bronze < Silver < Gold carries an ordering that a model can exploit with a single integer column. One-hot encoding erases that ordering and spends three columns to convey less information.

Tree-based models

A decision tree splits one column at a time. With one-hot encoding, a single informative categorical feature becomes many binary columns, and each split can only separate one level from the rest — so the tree needs greater depth to express what a single multi-way categorical split would capture, and under column subsampling (colsample_bytree, max_features) the individual indicators are often not even sampled. LightGBM and CatBoost handle categorical columns natively and generally do better with the raw or index-encoded column.

Features that are really identifiers

A near-unique column such as order_id produces one indicator per row. It cannot generalise and should be dropped, not encoded.

---

Distributed Categorical Pipelines in PySpark ML

In Apache Spark ML, categorical transformation is typically executed in three sequential stages:

1. StringIndexer: Maps string categorical values to numerical index columns ordered by category frequency (the most frequent category receives index 0.0).
2. OneHotEncoder: Converts category indices into binary vector representations (SparseVector).
3. VectorAssembler: Combines multiple numeric features and one-hot encoded vectors into a single unified features vector column required by Spark ML estimators.

[Raw Strings] ──> StringIndexer ──> [Category Indices] ──> OneHotEncoder ──> [Sparse Vectors] ──> VectorAssembler ──> [features Vector]

Handling Unseen Categories at Inference Time (handleInvalid)

When a model deployed in production encounters a category string that was never observed during training, StringIndexer must know how to respond. The handleInvalid parameter controls this behavior:

• handleInvalid="error" (Default in some versions): Throws a runtime JVM exception and halts execution. Unsafe for real-time production pipelines.
• handleInvalid="skip": Filters out and discards the entire row containing the invalid/unseen category. Useful for batch training, but dangerous in real-time inference endpoints.
• handleInvalid="keep" (Recommended for Production): Maps all unseen/invalid categories into a dedicated extra bucket (index equal to the number of distinct training categories). This allows downstream one-hot encoders and models to score new categories gracefully.

Complete End-to-End PySpark ML Pipeline

from pyspark.ml import Pipeline
from pyspark.ml.feature import StringIndexer, OneHotEncoder, VectorAssembler

categorical_cols = ["contract_type", "payment_method", "internet_service"]
numeric_cols = ["age", "tenure_months", "monthly_charges"]

# Step 1: Index categorical strings to frequency-ranked numbers
indexers = [
    StringIndexer(
        inputCol=c,
        outputCol=f"{c}_idx",
        handleInvalid="keep",             # Keep unseen categories in separate bucket
        stringOrderType="frequencyDesc"   # Most frequent category = index 0.0
    )
    for c in categorical_cols
]

# Step 2: One-Hot Encode indexed columns into SparseVectors
encoder = OneHotEncoder(
    inputCols=[f"{c}_idx" for c in categorical_cols],
    outputCols=[f"{c}_vec" for c in categorical_cols],
    dropLast=True,                        # Drop last category to avoid dummy variable trap
    handleInvalid="keep"
)

# Step 3: Assemble all continuous and encoded vector columns into single features column
assembler = VectorAssembler(
    inputCols=numeric_cols + [f"{c}_vec" for c in categorical_cols],
    outputCol="features",
    handleInvalid="keep"
)

# Step 4: Construct and fit the unified Pipeline
pipeline = Pipeline(stages=indexers + [encoder, assembler])
pipeline_model = pipeline.fit(train_df)

# Transform train and test splits
train_prepared = pipeline_model.transform(train_df)
test_prepared = pipeline_model.transform(test_df)

The dropLast Parameter: In PySpark's OneHotEncoder, dropLast=True (default) drops the last category entry, producing a vector of length $k-1$ for $k$ categories. This prevents the dummy variable trap (exact multi-collinearity) in unregularized linear models. For tree-based models or regularized linear models, dropLast=False can be used.

---

Feature Scaling Techniques & Formulations

Continuous features often exhibit drastically different numerical ranges (e.g., age between 18 and 90 vs. annual_income between $20,000 and $500,000). Features with large magnitudes can dominate distance metrics and gradient steps.

Raw Features:         Age: [18 ── 90]               Income: [$20,000 ──────────── $500,000]
StandardScaler:       Age: [-1.8 ── +2.1] (mean=0, std=1)    Income: [-1.5 ── +3.2] (mean=0, std=1)
MinMaxScaler:         Age: [0.0 ── 1.0]             Income: [0.0 ──────────────── 1.0]
RobustScaler:         Age: [-1.2 ── +1.5] (median/IQR)       Income: [-0.9 ── +2.4] (median/IQR)

Mathematical Formulations

1. Standardization (StandardScaler): $z = \frac{x - \mu}{\sigma}$ Centers features at $\mu = 0$ with unit variance $\sigma = 1$. Does not bound values into a fixed range. Outliers are preserved. PySpark Note: StandardScaler(withMean=True, withStd=True). Setting withMean=True on sparse vector inputs converts them to dense vectors, which can consume massive memory.
2. Min-Max Normalization (MinMaxScaler): $x_{scaled} = \frac{x - x_{min}}{x_{max} - x_{min}} \times (\text{max} - \text{min}) + \text{min}$ Compresses all feature values strictly into $[0.0, 1.0]$ (or custom bounds). Highly sensitive to extreme outliers, which compress standard observations into a narrow band.
3. Robust Scaling (RobustScaler): $x_{scaled} = \frac{x - \text{median}}{\text{IQR}}$ Uses median and Interquartile Range ($Q3 - Q1$). Resilient to outliers because centering and scaling metrics are not influenced by extreme tails.

---

Algorithm Scaling Sensitivity Comparison