3.1 Selecting the Appropriate Algorithm for a Scenario

3.1 Selecting the Appropriate Algorithm for a Scenario

Selecting the optimal supervised learning algorithm is a foundational architectural decision in enterprise machine learning workflows on Databricks. The choice directly influences model interpretability, training duration, computational cost, memory footprint, and real-time inference latency. Machine learning practitioners must balance statistical expressiveness against operational constraints, selecting the right algorithm and execution paradigm (single-node vs. distributed Spark ML) for their specific dataset and business objective.

---

Supervised Learning Paradigms: Regression vs. Classification

Supervised learning tasks map input feature vectors $\mathbf{x} \in \mathbb{R}^d$ to a known ground-truth label $y$. The mathematical structure of the label determines the paradigm:

+-----------------------------------------------------------------------------+
|                   SUPERVISED LEARNING PROBLEM TAXONOMY                      |
|                                                                             |
|   +--------------------------------+   +--------------------------------+   |
|   |           REGRESSION           |   |         CLASSIFICATION         |   |
|   |   Continuous Numeric Target    |   |   Discrete Categorical Target  |   |
|   |      y in (-inf, +inf) or R+   |   |        y in {0, 1} or {C_k}    |   |
|   +--------------------------------+   +--------------------------------+   |
|   | Loss Functions:                |   | Loss Functions:                |   |
|   | - Mean Squared Error (MSE / L2)|   | - Binary Cross-Entropy / LogLoss|  |
|   | - Mean Absolute Error (MAE / L1)|  | - Multi-Class Cross-Entropy   |   |
|   | - Huber / Smooth L1 Loss       |   | - Hinge Loss (SVM)             |   |
|   +--------------------------------+   +--------------------------------+   |
|   | Examples:                      |   | Examples:                      |   |
|   | - Customer Lifetime Value ($)  |   | - Credit Card Fraud (0 vs 1)   |   |
|   | - Demand Forecasting (Units)   |   | - Churn Prediction (Yes / No)  |   |
|   | - Machine Remaining Life (Days)|   | - Document Category (1 of K)   |   |
|   +--------------------------------+   +--------------------------------+   |
+-----------------------------------------------------------------------------+

Key Characteristics of Problem Types

1. Binary Classification: Target $y \in {0, 1}$. Models output predicted class probabilities $\hat{p} = P(y=1|\mathbf{x}) \in [0, 1]$ via a sigmoid activation $\sigma(z) = \frac{1}{1 + e^{-z}}$. Decisions are made by applying a decision threshold $\tau$ (default $0.5$).
2. Multiclass Classification: Target $y \in {1, 2, \dots, K}$ with mutually exclusive classes. Models output probability distributions over $K$ classes using the softmax function $\text{Softmax}(z_i) = \frac{e^{z_i}}{\sum_{j=1}^K e^{z_j}}$.
3. Continuous Regression: Target $y \in \mathbb{R}$. Predictions $\hat{y} = f(\mathbf{x})$ directly estimate the expected value $\mathbb{E}[y|\mathbf{x}]$, evaluated through residual metrics (RMSE, MAE, $R^2$).

---

Comprehensive Algorithm Taxonomy & Core Mechanics

Understanding the mathematical foundation of candidate algorithms allows engineers to diagnose model behavior, feature sensitivity, and failure modes.

+-----------------------------------------------------------------------------+
|                        SUPERVISED ALGORITHM TAXONOMY                        |
|                                                                             |
|   LINEAR MODELS              TREE-BASED MODELS        ENSEMBLE ARCHITECTURES|
|   +-------------------+      +-----------------+      +-----------------+   |
|   | Linear Regression |      | Decision Tree   |      | Random Forest   |   |
|   | - OLS (Unreg.)    |      | - CART          |      | (Bagging +      |   |
|   | - Ridge (L2)      |      | - Gini / Entropy|      | Feature Subspace|   |
|   | - Lasso (L1)      |      | - Variance Red. |      | Sampling)       |   |
|   | - ElasticNet      |      +--------+--------+      +-----------------+   |
|   |                   |               |                                     |
|   | Logistic Regr.    |               +-------------> +-----------------+   |
|   | - Binomial        |                               | Gradient Boosted|   |
|   | - Multinomial     |                               | Trees (GBT)     |   |
|   +-------------------+                               | - XGBoost       |   |
|                                                       | - LightGBM      |   |
|                                                       | - CatBoost      |   |
|                                                       | - Spark GBT     |   |
|                                                       +-----------------+   |
+-----------------------------------------------------------------------------+

Linear Models

• Ordinary Least Squares (OLS): Solves $\min_{\mathbf{w}} |\mathbf{y} - \mathbf{X}\mathbf{w}|^2_2$. Unregularized linear regression is analytically computed via the normal equation $\mathbf{w} = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}$. If features are highly correlated (multicollinearity), $\mathbf{X}^T\mathbf{X}$ becomes non-invertible or ill-conditioned, leading to extreme variance in weight estimates.
• Ridge Regression (L2 Regularization): Adds a quadratic weight penalty: $\min_{\mathbf{w}} |\mathbf{y} - \mathbf{X}\mathbf{w}|^2_2 + \lambda |\mathbf{w}|^2_2$. Ridge shrinks coefficients toward zero without forcing them exactly to zero. It stabilizes matrix inversion by solving $(\mathbf{X}^T\mathbf{X} + \lambda \mathbf{I})^{-1}\mathbf{X}^T\mathbf{y}$, effectively mitigating multicollinearity.
• Lasso Regression (L1 Regularization): Adds an absolute weight penalty: $\min_{\mathbf{w}} |\mathbf{y} - \mathbf{X}\mathbf{w}|^2_2 + \lambda |\mathbf{w}|_1$. Due to the sharp diamond geometry of the L1 ball at zero axes, Lasso drives non-informative feature coefficients to exact zero, performing automatic embedded feature selection.
• ElasticNet: Combines L1 and L2 penalties via a convex combination parameter $\alpha \in [0, 1]$: $\lambda [\alpha |\mathbf{w}|_1 + \frac{1-\alpha}{2} |\mathbf{w}|^2_2]$. When $\alpha=1$, ElasticNet is equivalent to Lasso; when $\alpha=0$, it is equivalent to Ridge. In PySpark ML, regParam defines $\lambda$ and elasticNetParam defines $\alpha$.
• Logistic Regression: Models the log-odds (logit) of a binary outcome: $\ln\left(\frac{p}{1-p}\right) = \mathbf{w}^T\mathbf{x} + b$. Trained using maximum likelihood estimation (MLE) minimizing binary cross-entropy loss. In PySpark ML, LogisticRegression supports binomial classification, multinomial classification, and ElasticNet regularization.

Decision Trees

Decision trees partition the feature space into orthogonal hyper-rectangles using recursive binary splitting. At each internal node, the algorithm evaluates all candidate split points across all features to maximize criterion purity gain:

• Classification Splitting Criteria:
  • Gini Impurity: $I_G(p) = 1 - \sum_{i=1}^C p_i^2$
  • Entropy (Information Gain): $H(p) = -\sum_{i=1}^C p_i \log_2(p_i)$
• Regression Splitting Criteria: Variance reduction / Mean Squared Error reduction.
• Behavior: Decision trees are non-parametric, handle non-linear boundaries naturally, and require no feature scaling. However, unconstrained decision trees have high variance and readily overfit training noise.

Random Forests (Bagging Ensemble)

Random Forests reduce the variance of individual decision trees through two orthogonal randomization mechanisms:

1. Bootstrap Aggregation (Bagging): Each tree is trained on an independently drawn bootstrap sample (sample with replacement of size $N$) from the training set. Roughly 63.2% of unique records appear in each bootstrap sample; the remaining 36.8% form the Out-of-Bag (OOB) evaluation set.
2. Feature Subspace Sampling: At every internal node split, only a random subset $m \ll d$ of candidate features is evaluated (typically $m = \sqrt{d}$ for classification, $m = d/3$ for regression). This decorrelates individual trees, ensuring that strong dominant features do not dictate every tree's initial splits.

• Inference: Class predictions are aggregated via majority voting; regression predictions are averaged across all trees.

Gradient Boosted Trees (Sequential Additive Boosting)

Unlike Random Forests where trees are constructed independently in parallel, Gradient Boosted Trees construct trees sequentially in an additive stage-wise manner:

$$F_m(\mathbf{x}) = F_{m-1}(\mathbf{x}) + \eta \cdot h_m(\mathbf{x})$$

where $h_m(\mathbf{x})$ is a base weak learner (shallow decision tree) fit to the pseudo-residuals (negative gradients of the loss function $\mathcal{L}(y, F_{m-1}(\mathbf{x}))$), and $\eta \in (0, 1]$ is the shrinkage (learning rate) parameter.

• Key GBT Implementations on Databricks:
  • XGBoost (xgboost.XGBClassifier / XGBRegressor): Utilizes second-order Taylor expansion of the loss function (gradient and hessian), exact and approximate quantile split finding, built-in L1/L2 tree complexity regularization, and sparsity-aware split routing for missing values.
  • LightGBM (lightgbm.LGBMClassifier / LGBMRegressor / synapse.ml.lightgbm): Implements Gradient-based One-Side Sampling (GOSS) to filter small-gradient instances and Exclusive Feature Bundling (EFB) to bundle mutually exclusive sparse features. Employs leaf-wise (best-first) tree growth rather than depth-wise, achieving superior training speed and lower memory usage.
  • CatBoost: Implements ordered boosting to combat target leakage in small datasets and provides native, high-performance categorical encoding using target statistics calculated over permutation histories.
  • Spark ML GBT (GBTClassifier / GBTRegressor): Native distributed GBT implementation within pyspark.ml. Implements depth-wise boosting across distributed partitions.

---

Algorithm Tradeoff Evaluation Matrix

The table below details the performance, operational, and diagnostic tradeoffs across major supervised learning algorithm families:

---

Single-Node vs. Distributed ML Architecture Selection

A critical Databricks Machine Learning Associate competency is recognizing when to execute training on a single virtual machine versus scaling to a multi-node Spark ML cluster.

+-----------------------------------------------------------------------------+
|                   SINGLE-NODE VS. DISTRIBUTED ML DECISION                   |
|                                                                             |
|                           DATASET SIZE ON DISK/RAM                          |
|                                      |                                      |
|                 +--------------------+--------------------+                 |
|                 |                                         |                 |
|       Data Fits in Driver Memory              Data Exceeds Single Machine   |
|        (e.g., < 50 GB - 100 GB)                 RAM (e.g., > 100 GB to TBs) |
|                 |                                         |                 |
|                 v                                         v                 |
|   +---------------------------+             +---------------------------+   |
|   | SINGLE-NODE ML LIBRARIES  |             | DISTRIBUTED SPARK ML      |   |
|   | - Scikit-learn            |             | - pyspark.ml (Spark MLlib)|   |
|   | - Single-Node XGBoost     |             | - Distributed GBT/RF      |   |
|   | - LightGBM / CatBoost     |             | - SynapseML (Spark LightGBM)  |
|   +---------------------------+             +---------------------------+   |
|   | Architecture Benefits:    |             | Architecture Benefits:    |   |
|   | - Zero network shuffle    |             | - Out-of-core scalability |   |
|   | - Ultra-fast iterations   |             | - In-memory cluster data  |   |
|   | - Full Hyperopt/Optuna    |             | - Native Delta Lake scans |   |
|   | - GPU acceleration (CUDA) |             | - Petabyte-scale pipelines|   |
|   +---------------------------+             +---------------------------+   |
+-----------------------------------------------------------------------------+

Detailed Selection Criteria

1. Choose Single-Node ML (Scikit-learn, XGBoost, LightGBM) When:

   • The aggregated training dataset (features + labels) comfortably fits inside single-node driver or single-worker memory (e.g., after feature aggregation, sampling, or filtering).
   • Model exploration demands rich algorithmic options not present in Spark ML (e.g., specialized loss functions, CatBoost native categorical handling, complex neural architectures).
   • Single-node training achieves faster wall-clock execution because it avoids Spark distributed network shuffles, serialization overhead, and partition scheduling latency.
   • Hyperparameter search is scaled across the cluster using hyperopt.SparkTrials, evaluating multiple single-node models concurrently across worker nodes.

2. Choose Distributed Spark ML (pyspark.ml) When:

   • The raw training dataset is hundreds of gigabytes or terabytes residing in distributed Delta tables, making driver memory collection (toPandas()) impossible (causing Out-Of-Memory / OOM crashes).
   • Feature transformations and modeling must execute within a unified, distributed pipeline (pyspark.ml.Pipeline) directly on Spark cluster executors.
   • End-to-end batch ETL and scoring workflows operate natively on Spark DataFrames without moving data out of the distributed Lakehouse boundary.

---

Implementation Code Examples

Single-Node XGBoost with Scikit-learn API

import xgboost as xgb
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score

# Convert sampled Spark DataFrame to Pandas if memory permits
pdf = spark.table("lakehouse_gold.customer_features").sample(0.1, seed=42).toPandas()

X = pdf.drop(columns=["customer_id", "churned"])
y = pdf["churned"]

X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)

# Instantiate and train single-node XGBoost
model = xgb.XGBClassifier(
    n_estimators=300,
    learning_rate=0.05,
    max_depth=5,
    subsample=0.8,
    colsample_bytree=0.8,
    reg_alpha=0.1,
    reg_lambda=1.0,
    random_state=42,
    eval_metric="logloss"
)

model.fit(
    X_train, y_train,
    eval_set=[(X_val, y_val)],
    verbose=False
)

val_preds = model.predict_proba(X_val)[:, 1]
print(f"Validation ROC-AUC: {roc_auc_score(y_val, val_preds):.4f}")

Distributed PySpark ML GBT Classifier

from pyspark.ml.classification import GBTClassifier
from pyspark.ml.feature import VectorAssembler
from pyspark.ml.evaluation import BinaryClassificationEvaluator

# Load distributed Delta Lake table
df = spark.table("lakehouse_gold.customer_features")

feature_cols = ["age", "total_spend", "login_count", "support_tickets"]
assembler = VectorAssembler(inputCols=feature_cols, outputCol="features")
prepared_df = assembler.transform(df)

train_df, test_df = prepared_df.randomSplit([0.8, 0.2], seed=42)

# Distributed Gradient Boosted Tree Classifier
gbt = GBTClassifier(
    featuresCol="features",
    labelCol="churned",
    maxIter=50,
    maxDepth=5,
    stepSize=0.05,
    seed=42
)

gbt_model = gbt.fit(train_df)
predictions = gbt_model.transform(test_df)

evaluator = BinaryClassificationEvaluator(
    labelCol="churned",
    rawPredictionCol="rawPrediction",
    metricName="areaUnderROC"
)

auc = evaluator.evaluate(predictions)
print(f"Distributed Spark GBT ROC-AUC: {auc:.4f}")