3.6 Big Data Projects: Wrangling, Exploration, Feature Engineering & Model Training

3.6 Big Data Projects: Wrangling, Exploration, Feature Engineering & Model Training

Blueprint note: Big Data Projects is a separate Level II learning module from Machine Learning. Its learning outcomes are the steps of a data analysis project, data preparation and wrangling, data exploration, feature extraction and selection from text, model training, and evaluating the fit of a machine learning algorithm. Vignettes describe a project and ask which step is being described, what went wrong, or how a printed confusion matrix should be read.

---

1. The Five Steps of a Financial Data Analysis Project

The two branches differ mainly at step 3. Everything after tokenization is common.

---

2. Data Preparation and Wrangling: Structured Data

Cleansing addresses five defect types:

1. Incompleteness — missing values; the remedy is omission, or imputation with the mean, median, mode, or a model-based estimate.
2. Invalidity — values outside a permissible range, such as a negative price.
3. Inaccuracy — values that are valid in form but wrong in fact.
4. Inconsistency — values that conflict with other fields, such as a country field that contradicts a currency field.
5. Non-uniformity — the same quantity expressed in different formats, such as mixed date conventions or mixed currency units.
6. Duplication — repeated records that overweight an observation.

Preprocessing then transforms clean data:

• Extraction — deriving a new variable from existing ones, such as age from date of birth.
• Aggregation — combining related variables into one.
• Filtration — deleting rows not needed for the project.
• Selection — deleting columns not needed.
• Conversion — putting variables into consistent types: nominal, ordinal, integer, ratio.

Outlier handling uses two named techniques. Trimming (truncation) removes the extreme observations entirely — a 2% trim removes the highest and lowest 1%. Winsorization replaces extreme values with the value at a stated percentile, keeping the observation count intact. Winsorization is usually preferred when the sample is small, because trimming discards information.

Scaling puts variables on comparable ranges, which matters for penalized regression, support vector machines, and k-nearest neighbour:

• Normalization rescales to $[0,1]$: $X_{norm} = (X - X_{min}) / (X_{max} - X_{min})$. It is sensitive to outliers and does not assume normality.
• Standardization centres and scales to a mean of 0 and standard deviation of 1: $Z = (X - \mu) / \sigma$. It assumes an approximately normal distribution and is less sensitive to outliers.

---

3. Text Preparation and Wrangling

Cleansing the raw text removes:

• html tags and other markup;
• punctuation, though some marks are replaced with annotations because they carry meaning — a percentage sign becomes a token such as percentSign, and currency symbols and question marks are similarly annotated rather than simply deleted;
• numbers, replaced with an annotation such as number when the magnitude is irrelevant to the task;
• extra white space, tabs, and line breaks.

Normalization then applies four operations:

1. Lowercasing, so that "Growth" and "growth" are one token.
2. Removing stop words — high-frequency, low-information words such as the, is, a. Financial applications sometimes retain negations, because "not profitable" and "profitable" must remain distinct.
3. Stemming — chopping to a root form by rule: analysing, analysed, and analyses all become analys. It is fast and crude.
4. Lemmatization — mapping to a dictionary base form using context: better becomes good. It is more accurate and computationally more expensive.

Tokenization splits the normalized text into tokens, and bag-of-words (BOW) represents a document as its unordered collection of tokens. N-grams preserve limited word order: a bigram treats "interest rate" as a single token, which matters because the unigrams "interest" and "rate" separately lose the concept.

The output is a document term matrix (DTM): rows are documents, columns are tokens, and each cell holds the frequency of that token in that document. A DTM built from a large corpus is extremely sparse — most cells are zero — which is exactly why feature selection is necessary.

4. Data Exploration: Feature Selection and Feature Engineering

Exploratory data analysis (EDA) comes first: histograms, box plots, scatter plots, and word clouds for text. Its purposes are to understand distributions, to spot relationships worth modelling, and to reveal data problems that cleansing missed.

Feature selection reduces the number of features, cutting noise and the risk of overfitting. For text, the standard tools are:

• Term frequency (TF) filters. Tokens with very high TF across the whole corpus carry little discriminating power and are dropped along with stop words; tokens with very low TF appear too rarely to generalise and are also dropped. The useful vocabulary sits in the middle.
• Document frequency (DF) — the proportion of documents containing the token.
• Chi-square statistic — ranks tokens by their association with the target class; the highest-ranked tokens are retained.
• Mutual information (MI) — measures how much information a token contributes about the class. MI is 0 when a token is equally distributed across classes and approaches 1 when it appears in only one class.

Feature engineering creates new, more informative features:

• converting numbers to categories such as decile buckets;
• n-grams as described above;
• name entity recognition (NER), tagging tokens as organisation, person, location, money, or date;
• parts of speech (POS) tagging, which distinguishes the noun "lead" from the verb "lead".

The two activities pull in opposite directions and must be balanced: too few features underfit, too many overfit and slow training.

---

5. Model Training and Evaluating the Fit

Model training has three components that are iterated, not performed once:

1. Method selection — supervised or unsupervised, linear or non-linear, driven by the data type, the size of the dataset, and whether interpretability is required.
2. Performance evaluation — measuring fit on data the model has not seen.
3. Tuning — adjusting hyperparameters to move along the bias-variance trade-off.

The confusion matrix and its four metrics

For a binary classifier, compare predictions against actual classes:

• Precision $= TP / (TP + FP)$ — of everything flagged, how much was real. High precision matters when a false alarm is costly.
• Recall (sensitivity) $= TP / (TP + FN)$ — of everything real, how much was caught. High recall matters when a miss is costly, as in fraud or default screening.
• Accuracy $= (TP + TN) / (TP + FP + TN + FN)$.
• F1 score $= (2 \times \text{precision} \times \text{recall}) / (\text{precision} + \text{recall})$ — the harmonic mean.

F1 is the preferred metric under class imbalance. If 2% of loans default, a model that predicts "no default" for everything achieves 98% accuracy, zero recall, and an F1 of 0. Accuracy alone would call that model excellent.

Worked evaluation

A model screening 1,000 loans for default produces TP = 60, FP = 40, FN = 30, TN = 870.

• Precision $= 60/(60+40) = 0.600$
• Recall $= 60/(60+30) = 0.667$
• Accuracy $= (60+870)/1{,}000 = 0.930$
• F1 $= 2(0.600)(0.667)/(0.600+0.667) = 0.8004/1.267 = 0.632$

Accuracy of 93.0% looks strong, but F1 of 0.632 shows the model misses a third of actual defaults. Lowering the classification threshold would raise recall and lower precision; where to set it depends on the relative cost of a missed default versus an unnecessary review.

Tuning and the fitting curve

• Bias error is the in-sample error of an underfitted model; variance error is the out-of-sample error of an overfitted model. Total error is minimised at an intermediate model complexity.
• A fitting curve plots error against a complexity or regularization parameter, and the optimum sits where the validation error stops falling and begins to rise.
• Regularization (the penalty term $\lambda$ in LASSO or ridge) reduces variance at the cost of some bias. Raising $\lambda$ shrinks coefficients and, in LASSO, sets some to exactly zero, performing feature selection automatically.
• K-fold cross-validation splits the data into k subsets, trains on $k-1$ and validates on the remaining one, rotating through all k. It uses the sample efficiently and mitigates the risk that a single arbitrary validation split flatters the model.

Ceiling analysis

Once a pipeline is running, ceiling analysis evaluates each component in turn to find where improvement pays best. If perfecting the tokenizer would raise overall accuracy by 1 percentage point but perfecting the feature-selection stage would raise it by 8, engineering effort belongs in feature selection.