2.3 Unsupervised Learning: Clustering & Dimensionality Reduction

Unsupervised Learning: Clustering & Dimensionality Reduction

Exam Tip: On the OCI AI Foundations Associate (1Z0-1122-26) exam, understand how DBSCAN differs from k-Means regarding cluster shape and outlier handling. Pay close attention to Principal Component Analysis (PCA) as a variance-maximizing linear projection, and know how to interpret Lift in association rule mining (a Lift value greater than 1 indicates a genuine positive correlation between items).

---

Defining Unsupervised Learning

Unsupervised Learning processes datasets that contain exclusively feature vectors $X = {x_1, x_2, \dots, x_n}$ without associated ground-truth target labels $y$. Without a teacher or labeled feedback signal to calculate prediction errors, the objective is to discover underlying structural properties, probability densities, latent representations, or natural groupings within the data.

Key enterprise applications include customer segmentation in marketing, fraud and intrusion detection in cybersecurity, seismic analysis in energy exploration, and preprocessing compression for downstream supervised tasks.

---

Clustering Techniques

Clustering is the unsupervised task of partitioning an unlabeled dataset into subgroups (clusters) such that observations assigned to the same cluster share high similarity, while observations assigned to distinct clusters exhibit high dissimilarity.

1. k-Means Clustering

k-Means is an iterative, centroid-based partitioning algorithm that divides $n$ observations into $k$ distinct, non-overlapping clusters.

Algorithm Mechanics:

1. Initialization: The practitioner selects the hyperparameter $k$ (number of clusters). The algorithm initializes $k$ centroids randomly within feature space (or strategically via k-Means++, which seeds initial centroids far apart to accelerate convergence and avoid poor local minima).
2. Assignment Step: Every observation is assigned to its nearest centroid, typically measured via Euclidean distance: $c^{(i)} = \arg\min_j \| x^{(i)} - \mu_j \|^2$
3. Update Step: Each centroid's coordinates are recalculated as the mathematical mean of all data points assigned to that cluster: $\mu_j = \frac{1}{|S_j|} \sum_{i \in S_j} x^{(i)}$
4. Convergence: Steps 2 and 3 repeat iteratively until centroid positions stabilize (shift below an epsilon threshold) or a maximum iteration limit is reached.

Optimization Objective: Minimizing the Within-Cluster Sum of Squares (WCSS), also referred to as cluster inertia: $\text{WCSS} = \sum_{j=1}^{k} \sum_{x \in S_j} \| x - \mu_j \|^2$

Determining the Optimal Number of Clusters ($k$):

• The Elbow Method: Plots WCSS against a range of sequential $k$ values (e.g., $k=1$ to $10$). As $k$ increases, WCSS naturally decreases. The optimal $k$ corresponds to the inflection point ("elbow") where the marginal decrease in WCSS slows noticeably.
• Silhouette Analysis: Measures how close each point in a cluster is to points in neighboring clusters. The Silhouette Coefficient $s(i)$ ranges from $-1.0$ to $+1.0$: $s(i) = \frac{b(i) - a(i)}{\max(a(i), b(i))}$ where $a(i)$ is the mean intra-cluster distance between point $i$ and all other points in its assigned cluster, and $b(i)$ is the mean nearest-cluster distance from point $i$. A score near $+1.0$ indicates well-separated, dense clusters; a score near $0.0$ indicates overlapping clusters; and negative scores suggest points assigned to the wrong cluster.

Limitations: k-Means requires pre-specifying $k$, is sensitive to outliers (which distort the mean), and assumes clusters are spherical, isotropic, and of equal variance.

2. Hierarchical Clustering

Hierarchical clustering builds a nested hierarchy of clusters visualized as a tree structure called a Dendrogram. By cutting the dendrogram horizontally at a chosen distance threshold, practitioners extract a specific number of clusters.

Hierarchical clustering operates via two distinct paradigms:

• Agglomerative (Bottom-Up): Begins with every observation treated as an individual singleton cluster ($n$ clusters). At each step, the two closest clusters are merged, repeating until all observations belong to a single global root cluster.
• Divisive (Top-Down): Begins with all observations grouped into one single cluster. The algorithm recursively splits clusters into smaller sub-clusters until every point forms its own singleton.

Linkage Criteria dictate how distance between two multi-point clusters is computed:

• Single Linkage: Minimum distance between any single point in cluster $A$ and any single point in cluster $B$ (prone to chaining artifacts).
• Complete Linkage: Maximum pairwise distance between points in $A$ and $B$ (produces compact, spherical clusters).
• Average Linkage: Average distance across all pairwise point combinations.
• Ward's Linkage: Minimizes the increase in total within-cluster variance resulting from merging the two clusters.

3. DBSCAN (Density-Based Spatial Clustering of Applications with Noise)

Unlike centroid-based or hierarchical methods, DBSCAN discovers clusters based on the local spatial density of observations, allowing it to identify clusters of arbitrary, complex geometric shapes while natively identifying outliers as noise.

DBSCAN relies on two primary hyperparameters:

• Epsilon ($\epsilon$): The radius defining the neighborhood surrounding any given data point.
• MinPts: The minimum number of points required within the $\epsilon$-neighborhood to designate that region as dense.

Point Classification:

• Core Point: Any point whose $\epsilon$-neighborhood contains at least $\text{MinPts}$ (including the point itself).
• Border Point: A point that falls within the $\epsilon$-neighborhood of a core point but contains fewer than $\text{MinPts}$ within its own radius.
• Noise / Outlier Point: Any point that is neither a core point nor a border point. These points are left unassigned, isolating anomalies natively.

Dimensionality Reduction & The Curse of Dimensionality

In enterprise machine learning, datasets frequently encompass hundreds or thousands of features (e.g., genomic sequencing, text bag-of-words, high-resolution sensor telemetry). This scale introduces severe theoretical and computational hurdles.

The Curse of Dimensionality

As the number of dimensional features $d$ increases, the volume of the feature space grows exponentially ($V \propto r^d$). This introduces three structural problems:

1. Data Sparsity: Training samples become isolated in vast empty space. The amount of training data required to maintain statistical sampling density grows exponentially with dimensionality.
2. Distance Metric Breakdown: In high-dimensional spaces, the Euclidean distance between the nearest neighbor and the furthest neighbor converges to nearly the same value. Consequently, distance-based algorithms (such as k-NN, k-Means, and SVM with RBF kernels) lose their discriminatory capability.
3. Overfitting & Latency: Large feature counts dramatically expand parameter counts, elevating overfitting risks while multiplying computational training and inference latency.

Principal Component Analysis (PCA)

Principal Component Analysis (PCA) is an unsupervised, linear dimensionality reduction technique. PCA projects high-dimensional data onto a lower-dimensional coordinate system defined by orthogonal (perpendicular) axes termed Principal Components (PCs).

• Mechanics:
  1. The data is centered by subtracting the mean of each feature (and standardized to unit variance if features possess differing physical scales).
  2. The algorithm calculates the covariance matrix of the standardized features.
  3. An eigenvalue decomposition (or Singular Value Decomposition / SVD) of the covariance matrix is computed:
     • The resulting eigenvectors represent the directional orientations of the new principal component axes.
     • The corresponding eigenvalues quantify the amount of variance captured along each principal component.
  4. The first principal component (PC1) accounts for the largest possible variance in the data. The second principal component (PC2) is strictly orthogonal to PC1 and captures the second largest variance, and so on.
  5. Practitioners choose a subset of $k$ principal components that preserve an acceptable threshold of cumulative explained variance (e.g., 90% or 95%), compressing the feature space from $d$ dimensions down to $k$ dimensions ($k \ll d$) while minimizing information loss.

Non-Linear Dimensionality Reduction for Visualization

While PCA is effective for linear compression, complex high-dimensional datasets often lie on non-linear manifolds:

• t-SNE (t-Distributed Stochastic Neighbor Embedding): A non-linear technique that maps high-dimensional points into a 2D or 3D visual space. It converts pairwise Euclidean distances into conditional probabilities and minimizes the Kullback-Leibler divergence between high-dimensional and low-dimensional representations using Student-t distributions. t-SNE excels at revealing local cluster structure but does not preserve global distances.
• UMAP (Uniform Manifold Approximation and Projection): A modern non-linear manifold learning algorithm based on Riemannian geometry. UMAP preserves both local neighborhood clusters and global data topology while executing significantly faster and scaling to larger sample sizes than t-SNE.

---

Association Rule Learning

Association Rule Learning is an unsupervised data mining paradigm designed to uncover strong relationships, affinities, and frequent co-occurrences among items within massive transactional datasets.

The classic application is Market Basket Analysis, which identifies purchasing patterns (e.g., "If a retail customer purchases bread and peanut butter, they are 80% likely to also purchase jelly"). Rules are formalized as implications:

$X \implies Y \quad (\text{where } X \text{ is the Antecedent and } Y \text{ is the Consequent})$

The Three Core Association Metrics

1. Support: $\text{Support}(X \implies Y) = \frac{\text{Frequency}(X \cup Y)}{N}$ Measures the proportion of total transactions $N$ that contain both itemsets $X$ and $Y$. It assesses the overall statistical popularity of the combination.

2. Confidence: $\text{Confidence}(X \implies Y) = \frac{\text{Support}(X \cup Y)}{\text{Support}(X)}$ Measures the conditional probability that a customer purchases itemset $Y$, given that they have already purchased itemset $X$. It quantifies the operational reliability of the rule.

3. Lift: $\text{Lift}(X \implies Y) = \frac{\text{Confidence}(X \implies Y)}{\text{Support}(Y)} = \frac{P(X \cap Y)}{P(X) \times P(Y)}$ Measures the strength of the association rule over what would be expected if $X$ and $Y$ were entirely statistically independent:

   • $\text{Lift} = 1.0$: $X$ and $Y$ are independent; purchasing $X$ has no effect on purchasing $Y$.
   • $\text{Lift} > 1.0$: Positive affinity; purchasing $X$ significantly increases the likelihood of purchasing $Y$.
   • $\text{Lift} < 1.0$: Negative association (substitutes); purchasing $X$ reduces the likelihood of purchasing $Y$.