3.4 Machine Learning, Deep Learning & Big Data Projects

Machine Learning in Investment Management

Machine learning (ML) provides data-driven algorithms that learn patterns directly from large, complex, and high-dimensional financial datasets without relying on rigid parametric assumptions.

                             ┌──────────────────────────────────────────────────────────┐
                             │                Machine Learning Taxonomy                 │
                             └────────────────────────────┬─────────────────────────────┘
                                                          │
         ┌────────────────────────────────────────────────┼────────────────────────────────────────────────┐
         ▼                                                ▼                                                ▼
┌────────────────────────┐                       ┌────────────────────────┐                       ┌────────────────────────┐
│  Supervised Learning   │                       │ Unsupervised Learning  │                       │     Deep Learning      │
│ - Target: Labeled Y    │                       │ - Target: None         │                       │ - Multi-layer ANNs     │
│ - Ridge (L2) & Lasso   │                       │ - PCA (Dimension Red.) │                       │ - CNN (Images/Spatial) │
│ - SVM, Trees, Ensembles│                       │ - K-Means & Hierarch.  │                       │ - RNN / LSTM (Sequence)│
└────────────────────────┘                       └────────────────────────┘                       └────────────────────────┘

---

1. Machine Learning Taxonomy

---

2. Supervised Learning: Penalized Regression & Classification

Penalized Linear Regression (Regularization)

Standard OLS minimizes $\sum (Y_i - \hat{Y}_i)^2$. When the feature count $k$ is large relative to $n$, OLS overfits the training data. Regularization introduces a penalty term scaled by hyperparameter $\lambda \ge 0$:

1. Ridge Regression ($L_2$ Regularization): $\min_{\beta} \left[ \sum_{i=1}^n (Y_i - X_i \beta)^2 + \lambda \sum_{j=1}^k b_j^2 \right]$
   • Shrinks coefficients asymptotically toward zero but never sets them exactly to zero.
   • Highly effective when features suffer from severe multicollinearity.
2. Lasso Regression ($L_1$ Regularization): $\min_{\beta} \left[ \sum_{i=1}^n (Y_i - X_i \beta)^2 + \lambda \sum_{j=1}^k |b_j| \right]$
   • Shrinks coefficients and sets non-essential coefficients exactly to zero as $\lambda$ increases.
   • Acts as an automatic feature selection tool, generating sparse and interpretable models.
3. Elastic Net: Combines both $L_1$ and $L_2$ penalties to handle groups of highly correlated features.

Classification Algorithms

• Support Vector Machines (SVM): Identifies an optimal decision boundary (hyperplane) that maximizes the margin between discrete classes. For non-linearly separable data, SVM maps features into higher-dimensional space via kernel functions (e.g., radial basis function - RBF).
• K-Nearest Neighbors (KNN): Non-parametric instance-based algorithm that classifies an observation based on the majority vote of its $k$ closest neighbors in feature space (using Euclidean or Manhattan distance). Requires feature normalization/standardization.
• Decision Trees: Partition feature space recursively into homogeneous rectangular regions using purity criteria (e.g., Gini Impurity or Entropy). Trees are non-parametric and intuitive but prone to overfitting unless pruned.
• Ensemble Methods:
  • Bagging (Bootstrap Aggregating): Fits multiple trees in parallel on randomly bootstrapped subsets of data and averages their predictions (e.g., Random Forests, which also randomly select a subset of features at each split). Primarily reduces variance without increasing bias.
  • Boosting: Builds trees sequentially, where each successive tree assigns greater weight to observations misclassified by prior trees (e.g., AdaBoost, Gradient Boosted Decision Trees - XGBoost, LightGBM). Primarily reduces bias and variance.

---

3. Unsupervised Learning: PCA & Clustering

Principal Component Analysis (PCA)

• Transforms $p$ correlated features into $p$ uncorrelated, orthogonal linear combinations called principal components ($PC_1, PC_2, \dots, PC_p$).
• $PC_1$ accounts for the maximum possible variance; each subsequent component explains the maximum remaining variance orthogonal to preceding components.
• In fixed income, the first 3 principal components typically capture over $95%$ of yield curve movements: Level ($PC_1$), Slope ($PC_2$), and Curvature ($PC_3$).

Clustering Techniques

• K-Means Clustering: Partitions $n$ observations into $K$ non-overlapping clusters by iteratively assigning points to the nearest cluster centroid and updating centroids. The optimal $K$ is selected via the Elbow Method (plotting within-cluster dispersion vs. $K$).
• Hierarchical Clustering: Builds a tree-like hierarchy (dendrogram). Can be agglomerative (bottom-up: starts with each point as its own cluster and merges closest pairs) or divisive (top-down).

---

4. Deep Learning Architectures

2026 currency note: the learning outcome covering neural networks, deep learning nets, and reinforcement learning was removed from the 2026 Level II Machine Learning module — the single curriculum change from 2025. The surviving outcome asks only that you describe supervised learning, unsupervised learning, and deep learning, so the material below is context rather than a testable calculation.

Deep Learning utilizes Artificial Neural Networks (ANNs) with multiple hidden layers between input and output layers:

• Multilayer Perceptron (MLP): Fully connected feedforward network with non-linear activation functions (e.g., ReLU, Sigmoid) trained via backpropagation and gradient descent.
• Convolutional Neural Networks (CNNs): Use convolution filters and pooling layers to extract spatial patterns. In finance, CNNs analyze satellite imagery (e.g., counting cars in retail parking lots, tracking oil tanker shadows) or financial chart patterns.
• Recurrent Neural Networks (RNNs) & LSTMs: Incorporate recurrent feedback loops and memory cells to capture sequential and temporal dependencies in financial time series and text narratives (e.g., central bank statements).

---

5. Model Evaluation & Performance Metrics

Bias-Variance Tradeoff & Cross-Validation

• Underfitting (High Bias): The model is overly simplistic, performing poorly on both training and test data.
• Overfitting (High Variance): The model fits training data noise, achieving near-perfect in-sample fit but terrible out-of-sample prediction.
• $K$-Fold Cross-Validation: Divides training data into $K$ equal subsets. In each iteration, $K-1$ folds train the model and the remaining fold validates it; performance is averaged across all $K$ folds to select optimal hyperparameters (e.g., $\lambda$).

Confusion Matrix & Classification Metrics

Key Metrics:

1. Accuracy: Overall fraction of correct predictions: $\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}$
2. Precision (Positive Predictive Value): Fraction of positive predictions that are truly positive (crucial when False Positives are costly, e.g., trading signal execution): $\text{Precision} = \frac{TP}{TP + FP}$
3. Recall / Sensitivity (True Positive Rate - TPR): Fraction of actual positives correctly identified (crucial when False Negatives are catastrophic, e.g., corporate default or fraud detection): $\text{Recall} = \frac{TP}{TP + FN}$
4. Specificity (True Negative Rate - TNR): Fraction of actual negatives correctly identified: $\text{Specificity} = \frac{TN}{TN + FP}$
5. F1-Score: The harmonic mean of Precision and Recall, providing a balanced evaluation under class imbalance: $\text{F1-Score} = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}} = \frac{2 \times TP}{2 \times TP + FP + FN}$
6. Receiver Operating Characteristic (ROC) & Area Under Curve (AUC): Plots TPR (Recall) on the y-axis against FPR ($1 - \text{Specificity}$) on the x-axis across all classification thresholds. An $\text{AUC} = 0.50$ represents random guessing, whereas $\text{AUC} = 1.00$ represents perfect discrimination.

---

6. Big Data in Investment Analysis & NLP Pipelines

The 4 Vs of Big Data

• Volume: Immense size of datasets (terabytes/petabytes).
• Velocity: High speed of incoming real-time streaming data.
• Variety: Heterogeneous formats (unstructured text, audio, images, structured logs).
• Veracity: Quality, reliability, and noise level in underlying data.

Alternative Data Sources in Asset Management

• Sensors & Satellite Imagery: Tracking supply chains, agricultural crop health, port congestion.
• Consumer Data: Aggregated credit card transactions, web traffic, app download statistics.
• Social Sentiment & Web Scraping: Real-time sentiment analysis from social media and consumer forums.

Natural Language Processing (NLP) Text Preparation Pipeline

Raw Text ──► Cleansing & Normalization ──► Tokenization ──► Stop-Word Removal ──► Lemmatization/Stemming ──► Feature Matrix (BoW / TF-IDF)

1. Text Cleansing & Normalization: Stripping HTML tags, punctuation, special characters, and converting all text to lowercase.
2. Tokenization: Splitting continuous text into individual word or phrase tokens.
3. Stop-Word Removal: Removing ubiquitous non-informative words (e.g., "the", "and", "is", "at").
4. Stemming vs. Lemmatization:
   • Stemming: Fast, rule-based truncation of word endings (e.g., "investing", "invested", "investor" $\to$ "invest"); may produce non-words.
   • Lemmatization: Dictionary-based morphological reduction to the true semantic root (e.g., "was", "is", "are" $\to$ "be").
5. $N$-Grams: Grouping contiguous sequences of $n$ words (e.g., bigrams "credit default", "cash flow") to preserve semantic context.
6. Feature Representation:
   • Bag-of-Words (BoW): Counts raw term occurrences across documents (Document-Term Matrix).
   • TF-IDF (Term Frequency-Inverse Document Frequency): Weights term $t$ in document $d$ relative to corpus frequency: $\text{TF-IDF}_{t,d} = \text{TF}_{t,d} \times \ln\left(\frac{N}{\text{DF}_t}\right)$ Where $N$ is total documents and $\text{DF}_t$ is the number of documents containing term $t$. Terms appearing frequently in a specific document but rarely across the entire corpus receive the highest weights.