8.3 Machine Learning and Prediction

Machine Learning and Prediction

QA–15 moves from ML taxonomy to the predictive toolkit: how models map features into forecasts, how penalties and trees control complexity, and how classification quality is scored when the cost of false alarms differs from missed events—central for credit default flags, fraud, and breach prediction.

Linear Regression for Prediction

In predictive mode, linear regression is still

ŷ = β₀ + β₁ x₁ + … + βₚ xₚ

but success is judged by out-of-sample MSE, MAE, or economic P&L of decisions based on ŷ—not only by t-stats. OLS minimizes training sum of squared errors. Strengths: fast, interpretable partial effects (within the linear class), strong baseline. Weaknesses: cannot capture interactions/nonlinearities unless you engineer features; sensitive to collinearity without penalties.

Worked predictive check

Train MSE = 1.0, test MSE = 1.1 → decent generalization for a linear model. Train MSE = 0.2, test MSE = 2.5 → overfit or regime shift; do not ship on train metrics alone.

Logistic Regression for Prediction

For binary Y ∈ {0,1} (default or not), logistic regression models

P(Y = 1 | x) = 1 / (1 + exp(−(β₀ + β′x))) = σ(β₀ + β′x)

where σ is the sigmoid. Coefficients are typically fit by maximum likelihood. Predictions can be reported as probabilities or thresholded (e.g., flag default if p̂ ≥ 0.5, or a risk-appetite cutoff like 0.10).

Log-odds interpretation: a unit rise in xⱼ shifts log-odds by βⱼ, holding other features fixed. For FRM prediction questions, emphasize calibration of probabilities and threshold choice under asymmetric costs—not only in-sample likelihood.

Encoding Categorical Variables

Algorithms need numbers. Categorical features (rating bucket, sector, region) must be encoded:

• One-hot / dummy encoding: create binary columns for categories. With an intercept, drop one baseline category to avoid the dummy variable trap (perfect multicollinearity).
• Ordinal encoding: map ordered tiers (AAA=1 … C=7)—imposes numeric spacing that may be wrong.
• Target / frequency encoding: replace category by historical mean y or frequency—powerful but leakage-prone if computed with full-sample including the row’s own y; compute on folds carefully.

Worked dummy example

Sector ∈ {Banks, Energy, Tech}. With intercept, include Banks and Energy dummies; Tech is baseline. Predicted mean for Tech uses β₀; Banks use β₀ + β_Banks. Including all three dummies plus intercept makes X′X singular for OLS.

Ridge Versus LASSO

When p is large or predictors are correlated, penalized linear/logistic models stabilize prediction.

Ridge (L2): minimize SSE + λ Σ βⱼ² (often not penalizing the intercept). Shrinks coefficients toward zero but rarely exactly to zero. Good for many correlated features (groups shrink together).

LASSO (L1): minimize SSE + λ Σ |βⱼ|. Shrinks and sets some coefficients exactly to zero—automatic feature selection. Can be unstable when predictors are highly correlated (picks one of a group arbitrarily).

Elastic net blends L1 and L2. λ (and mixing weights) are hyperparameters chosen by validation/cross-validation—not by maximizing in-sample R².

Worked penalty intuition

Unpenalized OLS coefficients on 50 noisy macro features: wild signs, test MSE high. Ridge with tuned λ shrinks everyone; test MSE drops. LASSO with tuned λ keeps 8 features and zeros 42; interpretability rises if the sparse set is stable across resamples.

Decision Trees

A decision tree (CART-style) recursively partitions the feature space with axis-aligned splits that minimize impurity (classification: Gini or entropy) or variance (regression). Prediction: average y (regression) or majority class / class probability (classification) in the terminal leaf.

Pros: captures nonlinearities and interactions; little need for rescaling; readable if shallow. Cons: high variance—small data changes can rewrite the tree; deep trees overfit.

Hyperparameters: max depth, min samples per leaf, min impurity decrease. Pruning or depth caps are the first defense against memorization.

Worked split sketch

Classify default. Root split: leverage > 0.45 versus ≤ 0.45. Left child (high leverage) splits on interest coverage < 2. Leaves report empirical default rates 25%, 8%, 3%, 1%. A new firm with leverage 0.50 and coverage 1.5 falls into the 25% leaf—transparent, but one deep tree may not generalize.

Ensembles: Bagging, Random Forests, Boosting

Bagging: train many models on bootstrap resamples; average predictions (or vote). Cuts variance of unstable learners (trees).

Random forest: bagged trees with random feature subsets at each split—decorrelates trees, often stronger than plain bagging.

Boosting (e.g., gradient boosting): add trees sequentially that focus on previous residuals/errors. Reduces bias aggressively; needs care with learning rate and depth to avoid overfit.

Ensembles win many tabular prediction contests in credit and fraud—but monitoring, explainability (SHAP, etc.), and stability under regime shifts remain risk-management requirements.

K-Nearest Neighbors and Support Vector Machines

KNN: predict y using the K closest training points in feature space (majority vote or average). Nonparametric and simple; sensitive to scale and to the curse of dimensionality; prediction cost grows with training-set size.

SVM (support vector machine): finds a separating hyperplane with maximum margin between classes (soft margin allows violations with a penalty C). Kernel SVMs (RBF, polynomial) implicitly map features to richer spaces for nonlinear boundaries. Conceptually: focus on support vectors—points that define the margin—rather than fitting every observation equally.

Worked KNN sketch

K = 3, standardized features. New borrower distances to labeled neighbors: default, default, perform. Vote → predict default. If K = 1, a single noisy neighbor can flip the call—hence validation of K.

Neural Networks

A feed-forward neural net stacks layers: each layer computes linear combinations of inputs plus bias, then applies a nonlinear activation (ReLU, sigmoid, tanh). Depth and width create flexible function classes. Training typically minimizes log-loss or MSE via gradient methods (backpropagation, Adam, etc.).

Regularization: weight decay (L2), dropout, early stopping on validation loss, data augmentation where relevant. For tabular FRM-style problems, nets are not automatically better than boosted trees; they shine with large data and unstructured inputs (raw text with deep NLP, images). Exam point: nets approximate complex nonlinear maps but require careful validation and are harder to interpret than logistic regression.

Confusion Matrix and Comparison Metrics

For a binary classifier, count:

Definitions:

• Accuracy = (TP + TN) / (TP + TN + FP + FN)—misleading under rare defaults.
• Precision (positive predictive value) = TP / (TP + FP)—of flagged defaults, how many true?
• Recall (sensitivity, true positive rate) = TP / (TP + FN)—of true defaults, how many caught?
• Specificity = TN / (TN + FP)—of non-defaults, how many cleared?
• F1 = harmonic mean of precision and recall = 2 · precision · recall / (precision + recall).

Worked comparison

1,000 loans, 50 true defaults (5% rate).

Model A: TP=30, FP=40, FN=20, TN=910. Precision = 30/70 ≈ 0.43; Recall = 30/50 = 0.60; Accuracy = 940/1000 = 0.94.

Model B: TP=10, FP=5, FN=40, TN=945. Precision = 10/15 ≈ 0.67; Recall = 10/50 = 0.20; Accuracy = 955/1000 = 0.955.

Model B has higher accuracy and precision but much worse recall—dangerous if missing defaults is costly. A naive “always predict non-default” gets 95% accuracy and zero recall. Always compare models on metrics aligned with the loss function, not accuracy alone.

ROC curves plot TPR (recall) versus FPR = FP/(FP+TN) across thresholds; AUC summarizes ranking quality. Precision–recall curves are often more informative under class imbalance.

Exam Synthesis

Choose linear/logistic baselines first; encode categoricals without the dummy trap; use ridge for shrinkage and LASSO for sparsity; prefer ensembles when a single tree overfits; remember KNN/SVM geometry and net flexibility; score classifiers with a confusion-matrix mindset when class imbalance and asymmetric costs matter. Prediction quality is an out-of-sample property—QA–15 rewards candidates who never confuse training fit with operational performance.