2.2 Supervised Learning: Classification, Regression & Core Algorithms

Supervised Learning: Classification, Regression & Core Algorithms

Exam Tip: On the OCI AI Foundations Associate (1Z0-1122-26) exam, questions frequently test your ability to select the correct evaluation metric for business scenarios with imbalanced classes. Understand why Recall is prioritized when False Negatives are costly (e.g., disease detection or fraud), why Precision matters when False Positives are expensive (e.g., spam filtering or customer suspension), and the structural differences between Random Forests (bagging) and Gradient Boosted Trees (boosting).

---

Defining Supervised Learning

Supervised Learning is the most widely deployed machine learning paradigm in enterprise environments. In supervised learning, the algorithm is provided with a training dataset consisting of paired input-output examples:

$\mathcal{D} = \{(x_1, y_1), (x_2, y_2), \dots, (x_n, y_n)\}$

Here, each input vector $x_i \in \mathbb{R}^d$ contains $d$ features representing an observation, and each $y_i$ is a known, ground-truth target label. The objective of the algorithm is to approximate an unknown mapping function $f: X \to Y$ such that $f(x) \approx y$, allowing the model to accurately predict the target label for novel, unobserved feature vectors.

Supervised learning tasks divide fundamentally into two categories based on the nature of the target variable: Classification and Regression.

---

Classification: Problem Types & Evaluation Metrics

In classification tasks, the target variable is discrete and qualitative. The model acts as a decision boundary, partitioning feature space into distinct categorical assignments.

Classification Variants

• Binary Classification: The target variable has exactly two mutually exclusive classes, conventionally encoded as 0 and 1 (negative and positive). Examples include determining whether an incoming transaction is fraudulent (1) or legitimate (0), predicting employee attrition, or email spam detection.
• Multiclass Classification: The target variable encompasses three or more mutually exclusive classes, where each observation belongs to exactly one category. Examples include sorting customer service emails into five operational departments, or classifying medical imagery into benign, malignant, or unclassifiable categories.
• Multilabel Classification: Observations can be assigned multiple non-exclusive labels simultaneously. For example, a single enterprise technical document might be tagged with Cloud Architecture, Security, and Database Optimization concurrently.

The Confusion Matrix

The foundational tool for evaluating binary classification is the Confusion Matrix, which tabulates model predictions against ground truth:

Quantitative Classification Metrics

From these four counts, data scientists derive standard evaluation metrics:

1. Accuracy: $\text{Accuracy} = \frac{\text{TP} + \text{TN}}{\text{TP} + \text{FP} + \text{TN} + \text{FN}}$

   • Strengths & Limitations: Accuracy measures the overall proportion of correct classifications. However, it is fundamentally misleading in imbalanced datasets. For example, in a financial fraud dataset where 99.8% of transactions are legitimate and 0.2% are fraudulent, a naive model that predicts every transaction is legitimate achieves 99.8% accuracy while failing to detect a single fraudulent transaction.

2. Precision (Positive Predictive Value): $\text{Precision} = \frac{\text{TP}}{\text{TP} + \text{FP}}$

   • Significance: Out of all instances the model predicted as positive, what proportion was truly positive? High precision is paramount when the business cost of a False Positive is severe. For example, in automated spam filtering, a false positive means a critical business email is routed to the trash folder and lost.

3. Recall / Sensitivity (True Positive Rate): $\text{Recall} = \frac{\text{TP}}{\text{TP} + \text{FN}}$

   • Significance: Out of all actual positive instances present in the data, what proportion did the model successfully capture? High recall is essential when the cost of a False Negative is catastrophic. In medical cancer screening or jet engine flaw detection, failing to detect a positive case can result in loss of life or catastrophic structural failure.

4. F1-Score: $\text{F1-Score} = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}$

   • Significance: The harmonic mean of Precision and Recall. Unlike the arithmetic mean, the harmonic mean heavily penalizes extreme imbalances between precision and recall, serving as the benchmark metric for imbalanced classification.

5. ROC-AUC (Receiver Operating Characteristic - Area Under the Curve):

   • The ROC curve plots the True Positive Rate (Recall) against the False Positive Rate ($\text{FPR} = \frac{\text{FP}}{\text{FP} + \text{TN}}$) across all possible decision thresholds from $0.0$ to $1.0$.
   • AUC (Area Under the Curve) summarizes this performance into a single scalar value between $0.0$ and $1.0$. An AUC of $0.5$ represents random guessing, whereas an AUC of $1.0$ indicates perfect separation across all probability thresholds regardless of chosen cutoff.

---

Regression: Problem Types & Evaluation Metrics

In regression tasks, the target variable is continuous, quantitative, and ordered. The model estimates a numerical output on an open or bounded numerical scale.

Typical regression use cases include predicting customer lifetime value, estimating hardware component temperatures, forecasting electric grid demand, and predicting housing sale prices.

Quantitative Regression Metrics

Let $y_i$ represent the actual target value, $\hat{y}_i$ represent the model's predicted value, and $\bar{y}$ represent the mean of actual values across $n$ samples:

1. Mean Absolute Error (MAE): $\text{MAE} = \frac{1}{n} \sum_{i=1}^{n} |y_i - \hat{y}_i|$

   • Properties: Measures the average magnitude of prediction errors without regard to direction. MAE scales linearly with error and is robust to extreme outliers.

2. Mean Squared Error (MSE): $\text{MSE} = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2$

   • Properties: Computes the average of squared differences. By squaring individual residuals, MSE heavily penalizes large errors, making it sensitive to outliers. Because units are squared (e.g., dollars squared), it lacks immediate intuitive interpretability.

3. Root Mean Squared Error (RMSE): $\text{RMSE} = \sqrt{\text{MSE}} = \sqrt{\frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2}$

   • Properties: The square root of MSE brings the error metric back into the exact same physical units as the target variable (e.g., dollars or degrees Celsius). Like MSE, it heavily penalizes large outlier errors, making it the industry standard benchmark for regression tasks.

4. R-Squared ($R^2$, Coefficient of Determination): $R^2 = 1 - \frac{\sum_{i=1}^{n} (y_i - \hat{y}_i)^2}{\sum_{i=1}^{n} (y_i - \bar{y})^2}$

   • Properties: Measures the proportion of variance in the dependent variable explained by the independent features in the model. Values range from $1.0$ (perfect explanatory fit) to $0.0$ (performance equivalent to naively predicting the mean $\bar{y}$), and can be negative if the model performs worse than the simple horizontal mean line.

---

Core Supervised Algorithms

1. Linear Regression & Logistic Regression

• Linear Regression: Models a linear relationship between continuous input features $X$ and a continuous scalar output $y$: $\hat{y} = w_1 x_1 + w_2 x_2 + \dots + w_d x_d + b = w^T x + b$ Parameters are optimized to minimize the residual sum of squares using Ordinary Least Squares (OLS) or Gradient Descent.

• Logistic Regression: Despite its name, Logistic Regression is a classification algorithm. It maps linear feature combinations into calibrated class probabilities between $0$ and $1$ using the non-linear Sigmoid (logistic) function: $P(Y=1|X) = \sigma(z) = \frac{1}{1 + e^{-z}}, \quad \text{where } z = w^T x + b$ A default decision threshold (conventionally $0.5$) converts the resulting probability into a discrete class assignment (1 if $\sigma(z) \ge 0.5$, else 0). Parameters are optimized using maximum likelihood estimation via log-loss (binary cross-entropy).

2. Decision Trees

A Decision Tree partitions feature space into recursive axis-aligned rectangular regions via a hierarchical flowchart structure composed of a root node, internal decision nodes (representing attribute threshold tests), branches (outcomes of tests), and leaf nodes (terminal class labels or regression averages).

At each node, the algorithm searches across all features to find the split that maximizes purity:

• Gini Impurity (Classification): Measures the probability that a randomly chosen element from the set would be incorrectly labeled if it were randomly labeled according to the distribution of labels in the subset: $I_G = 1 - \sum_{i=1}^{C} p_i^2$
• Information Gain / Entropy (Classification): Grounded in Shannon entropy, measuring the reduction in informational uncertainty achieved by splitting on a feature: $H(S) = - \sum_{i=1}^{C} p_i \log_2(p_i)$

While highly interpretable and requiring minimal data scaling, unconstrained decision trees are prone to severe overfitting.

3. Ensemble Learning: Random Forests & Gradient Boosted Trees

Ensemble learning combines predictions from multiple base models (weak learners) to construct a superior, robust aggregate predictor:

• Random Forest (Bagging - Bootstrap Aggregating): Fits hundreds of deep, fully-grown decision trees in parallel on independently sampled bootstrap subsets of the training data (sampling with replacement). At each split, trees are restricted to choosing from a random subset of features. Predictions are combined via majority voting (classification) or mean averaging (regression). Random Forests primarily reduce model variance without increasing bias.
• Gradient Boosted Trees (Boosting - e.g., XGBoost, LightGBM, CatBoost): Trains decision trees sequentially. Each successive tree is explicitly trained to predict the pseudo-residuals (errors) of the ensemble constructed so far. By focusing sequential computational capacity on previously misclassified observations, boosting primarily reduces bias, producing state-of-the-art accuracy on structured tabular datasets.

4. Support Vector Machines (SVM)

Support Vector Machines construct an optimal decision hyperplane that maximizes the margin—the physical distance between the dividing hyperplane and the nearest data points of any class, termed the Support Vectors.

When data is not linearly separable in its native feature space, SVM applies the Kernel Trick. Mathematical kernel functions (such as the Radial Basis Function / RBF or Polynomial kernel) implicitly project input vectors into a higher-dimensional space where a linear separating hyperplane can be constructed, without explicitly computing expensive coordinate transformations.

5. k-Nearest Neighbors (k-NN)

k-Nearest Neighbors (k-NN) is an instance-based, non-parametric, lazy learning algorithm. It does not construct an explicit internal model during a separate training phase; instead, it stores the entire training dataset in memory.

When a new query point arrives, k-NN calculates distance metrics (such as Euclidean distance $d(p, q) = \sqrt{\sum (p_i - q_i)^2}$ or Manhattan distance) between the query point and all stored training observations, identifies the $k$ closest neighbors, and outputs the majority class (classification) or mean value (regression). k-NN is exceptionally sensitive to feature scaling, necessitating thorough feature standardization during preprocessing.

---

Comparison: Supervised Learning Paradigms & Metrics