2.3 Exploratory Data Analysis & Evaluating a Data Set

What EDA Is For

The Content Outline asks candidates to import, manipulate, and evaluate data sets. "Evaluate" is exploratory data analysis, and on the project it is an explicit rubric item: the candidate must describe anomalous characteristics of the data and how they were addressed.

EDA answers four questions before any model is fitted:

1. Is the data what I was told it is? Row counts, date ranges, totals reconciled to a source.
2. What is broken? Missingness, impossible values, duplicate keys, thin levels.
3. What shape does each predictor have? Skew, spikes, natural groupings, monotone or U-shaped relationships to the target.
4. What should the model form be? Which distribution, which transformations, which variables to band, which interactions to test.

A GLM fitted before this work is done is a guess. A GLM fitted after it is a hypothesis.

Weight Everything by Exposure

This is the single largest difference between generic data science EDA and actuarial EDA. Insurance records do not carry equal evidence. A one-month policy carries one twelfth of the information of a one-year policy; a 200-vehicle fleet carries 200 times the information of a single car.

Therefore:

• The mean frequency for a group is sum(claims) / sum(exposure), never mean(claims / exposure).
• The mean severity for a group is sum(loss) / sum(claim count), weighted by claim count, not by policy.
• The mean pure premium for a group is sum(loss) / sum(exposure).
• Every one-way plot should carry exposure as bars on a secondary axis so the reader can see which points are credible.

[!WARNING] An unweighted average of claims / exposure is dominated by short-term policies, because a single claim on a 0.08-year policy produces an empirical frequency of 12.5. This is the fastest way to produce a plausible-looking one-way plot that is entirely noise.

The Univariate Pass

For every field, produce and read:

Sentinel values deserve special attention. A driver_age of 99 or 999, a years_licensed of -1, and a vehicle_year of 1900 are almost always encoded missingness rather than real observations, and they will distort a fitted slope badly if they are treated as numbers.

The Bivariate Pass

Once each field is understood alone, look at each against the target. The workhorse exhibit is the one-way plot:

1. Group the data by the predictor — by level for a categorical, by band for a continuous.
2. Within each group compute exposure, observed frequency, observed severity, and observed pure premium.
3. Plot the observed statistic as a line or point, with exposure as bars behind it.

Read it for four things:

• Direction and shape. Monotone increasing, decreasing, U-shaped (driver age is the classic U), or flat.
• Credibility. Bands with tiny exposure will bounce; do not read a trend from them.
• Non-linearity. A continuous predictor whose banded relationship curves tells you that entering it as a single linear term will not fit, and that a polynomial, piecewise-linear term, or banding is needed.
• Thin levels. Levels with too little exposure to support their own coefficient must be grouped before modelling.

For predictor-to-predictor relationships, a correlation matrix for continuous fields and a two-way exposure table for categorical pairs will reveal the near-duplication that later shows up as multicollinearity or aliasing.

The Distinctive Shape of Loss Data

Three features of P&C loss data drive modelling choices, and all three are visible in EDA:

A large point mass at zero. Most policies have no claim. A histogram of pure premium is therefore a spike at zero with a thin smear to the right. This is exactly the structure the Tweedie distribution is built for, and seeing it is the justification for choosing Tweedie over Gamma for a pure premium target.

Extreme right skew in severity. Individual claim amounts span several orders of magnitude. Plot severity on a log scale — on a linear scale, one $4 million claim compresses every other observation onto the axis. The skew is also why Gamma with a log link usually beats a normal linear model for severity.

Heterogeneous exposure. Already covered above, and the reason offsets and weights exist.

Exploratory Graphics vs. Presentation Graphics

Gelman and Unwin's assigned paper draws a distinction that PCPA cares about in both directions. Information visualisation is designed to attract attention and communicate a message to a broad audience; it prizes novelty, aesthetics, and a single clear takeaway. Statistical graphics are designed to reveal structure and check models; they prize small multiples, default axes, density of information, and the ability to show what is wrong.

EDA needs statistical graphics. Quick, ugly, numerous, disposable plots are correct at this stage. The project report needs the other kind, but only after the exploratory work has told you what there is to say. Reversing the order — polishing a chart before you know whether the pattern is credible — wastes the scarcest resource in a two-week window.

An EDA Checklist for the Project

• Reconcile row count, exposure and incurred loss to the source file.
• Missingness table for every field, by segment as well as in total.
• Exposure by level for every categorical; flag levels below a credibility floor.
• Percentile table for every continuous field; flag sentinels and impossible values.
• One-way exposure-weighted plot of the target against every candidate predictor.
• Correlation matrix of continuous predictors.
• A written note of every anomaly found and the action taken — this is the wording you will reuse for rubric criterion A-4.