5.3 Categorical Predictors, Base Levels & High-Cardinality Grouping

The High-Cardinality Challenge in P&C Insurance

In property and casualty insurance, some of the most powerful risk discriminators are discrete categorical variables with hundreds, thousands, or tens of thousands of unique levels. These high-cardinality categorical predictors are central to risk classification:

1. Geographic Classifiers: 5-digit United States ZIP codes contain over $42,000$ active levels. In homeowners insurance, geographic location captures critical exposure differentials: distance to coastline (hurricane hazard), brush density (wildfire risk), local fire department responsiveness, and municipal building code enforcement.
2. Vehicle Classifiers: Vehicle identification numbers (VIN) decode into Make, Model, Sub-model, and Trim, creating $3,000$ to $10,000$ distinct vehicle classifications. These capture engine performance, advanced driver assistance systems (ADAS), repair parts costs, and crashworthiness.
3. Commercial Business Classifiers: Workers' compensation class codes, North American Industry Classification System (NAICS), and Standard Industrial Classification (SIC) codes exceed $1,000$ categories, differentiating occupational hazards from clerical office work to high-rise structural steel erection.

The Failure of Traditional One-Hot (Dummy) Encoding

Standard statistical modeling encodes categorical predictors by creating $K - 1$ binary dummy variables for a factor with $K$ levels. When applied to high-cardinality features in GLMs, this approach fails completely:

                    HIGH-CARDINALITY ONE-HOT FAILURE MODES
                                       │
         ┌─────────────────────────────┼─────────────────────────────┐
         ▼                             ▼                             ▼
Dimensionality Explosion       Degrees of Freedom Loss        Empty / Sparse Cells
42,000 ZIP codes create        Consumes tens of thousands     Rural ZIPs with 1-2 policies
41,999 new columns in X;       of degrees of freedom;         yield infinite or undefined
matrix memory exhausts         drastically inflates variance  MLE coefficients (separation)

• Dimensionality Explosion: Encoding $42,000$ ZIP codes adds $41,999$ columns to the design matrix $\mathbf{X}$. Storing and manipulating this sparse matrix requires massive memory and slows IRLS convergence to a crawl.
• Loss of Degrees of Freedom: Estimating tens of thousands of parameters depletes model degrees of freedom ($n - p$), drastically inflating standard errors across all coefficients.
• Complete Separation & Extreme Sparse Cells: In casualty portfolios, many levels are sparsely populated. A rural ZIP code might have only $2$ insured policyholders with $0.8$ earned car-years and $0$ claims. In a Poisson frequency GLM, the maximum likelihood estimate for a category with zero claims diverges to $-\infty$:

$\hat{\beta}_{\text{sparse}} = \ln\left( \frac{0}{\text{exposure}} \right) = -\infty$

Conversely, if a rare vehicle trim with $1$ exposure suffered a catastrophic total loss, MLE estimates a massive positive coefficient. The model suffers from severe estimation instability.

Target Encoding (Mean Encoding) & The Target Leakage Hazard

To compress high-cardinality categories into a manageable representation, machine learning practitioners frequently employ target encoding (also known as mean encoding or likelihood encoding).

Conceptual Definition

Target encoding replaces each categorical level $c \in {1, 2, \dots, K}$ with the expected value of the target response $Y$ within that level:

$\hat{S}_c = \mathbb{E}[Y \mid X = c]$

• For a frequency model, $\hat{S}_c$ is the empirical claim frequency in level $c$: $\hat{S}_c = \frac{\sum_{i \in c} y_i}{\sum_{i \in c} e_i}$
• For a pure premium / loss cost model, $\hat{S}_c$ is the average loss per earned exposure in level $c$: $\hat{S}_c = \frac{\sum_{i \in c} \text{Loss}_i}{\sum_{i \in c} e_i}$

Target encoding replaces a $K$-level categorical variable with a single continuous 1D numeric feature. This preserves monotonic ordering with respect to the loss cost and consumes exactly one degree of freedom in a GLM instead of the $K-1$ that dummy coding would require.

The Fatal Flaw: Target Leakage and Noise Memorization

Despite its simplicity, naive target encoding is disastrously flawed because it induces severe target leakage.

If the sample mean $\hat{S}_c$ is calculated using the target response of the entire training dataset—including observation $i$ itself—the model leaks the response into the predictor:

$\hat{S}_{c(i)} = \frac{y_i + \sum_{j \in c, j \ne i} y_j}{n_c}$

Consider an auto insurance portfolio with a rare vehicle trim code $c = \text{Vintage_Trim_42}$ containing exactly $n_c = 1$ policyholder. Suppose this policyholder suffered a severe $80,000 claim ($y_i = 80,000$).

1. Under naive target encoding, this vehicle trim is assigned an encoded feature value of $\hat{S}_c = 80,000$.
2. A downstream GLM or tree ensemble immediately recognizes that whenever $\text{Feature} = 80,000$, the target is $80,000, assigning this single predictor a massive positive weight.
3. The model appears to achieve near-perfect in-sample fit ($R^2 \approx 1.0$, deviance drops drastically). In reality, the algorithm has simply memorized target noise.
4. When deployed on unseen prospective policyholders, future insureds driving Vintage Trim 42 are hit with absurdly uncompetitive premiums, while the model fails to predict losses on other risks.

Credibility-Weighted Target Encoding & Bühlmann Shrinkage

Decades before modern data science discovered the vulnerability of naive target encoding, casualty actuaries solved the problem of sparse classification cells using Bühlmann credibility theory.

Actuarial Credibility Formulation

To prevent small, sparse categories from memorizing noise, credibility-weighted target encoding shrinks the empirical sample mean of category $c$ toward the global portfolio average:

$\hat{S}_c^{\text{cred}} = Z_c \bar{y}_c + (1 - Z_c) \bar{y}_{\text{global}}$

Where:

• $\bar{y}c = \frac{\sum{i \in c} y_i}{\sum_{i \in c} e_i}$ is the empirical target mean of category $c$.
• $\bar{y}{\text{global}} = \frac{\sum{i=1}^N y_i}{\sum_{i=1}^N e_i}$ is the grand portfolio mean across all policies.
• $Z_c \in [0, 1]$ is the Bühlmann credibility factor.

The Bühlmann Credibility Factor

In classical Bühlmann credibility, the credibility weight assigned to the category's own experience is defined as:

$Z_c = \frac{n_c}{n_c + k}$

Where $n_c$ is the exposure (or policy count) in category $c$, and $k$ is the structural ratio of the Expected Value of the Process Variance (EPV) to the Variance of the Hypothetical Means (VHM):

$k = \frac{\mathbb{E}[\text{Var}(Y \mid \Theta)]}{\text{Var}(\mathbb{E}[Y \mid \Theta])} = \frac{\text{EPV}}{\text{VHM}}$

                 BÜHLMANN CREDIBILITY SHRINKAGE SPECTRUM

  Sparse Category (n_c << k)                    Dense Category (n_c >> k)
  ──────────────────────────                    ─────────────────────────
  Z_c ≈ 0                                       Z_c ≈ 1
  Encoded Value -> Global Mean                  Encoded Value -> Category Mean
  Safe, robust, prevents noise                  Reflects distinct credibility

         0.0 ├───────────────────────●───────────────────────┤ 1.0
            Z_c = 0                                         Z_c = 1
      (Pure Global Mean)                             (Pure Category Mean)

1. For Sparse Categories ($n_c \to 0$): As $n_c \ll k$, $Z_c \to 0$. The category's sparse, volatile empirical mean is discarded, and its encoded value reverts almost entirely to the reliable portfolio baseline $\bar{y}_{\text{global}}$. An isolated $80,000 claim in a 1-policy ZIP code is shrunk heavily back to the state average, preventing rate distortion.
2. For Dense Categories ($n_c \to \infty$): As $n_c \gg k$, $Z_c \to 1$. The category possesses full credibility, and its encoded value reflects its own robust empirical loss experience.

Out-of-Fold (OOF) Target Encoding Pipeline

Even with credibility shrinkage, target leakage can still occur if an observation's own target is included in $\bar{y}_c$. To achieve complete leakage isolation, actuaries combine credibility shrinkage with an Out-of-Fold (OOF) calculation architecture:

                      OUT-OF-FOLD (OOF) ENCODING PIPELINE

       Training Data (Partitioned into M = 5 Folds)
       ┌──────────┬──────────┬──────────┬──────────┬──────────┐
       │  Fold 1  │  Fold 2  │  Fold 3  │  Fold 4  │  Fold 5  │
       └────┬─────┴──────────┴──────────┴──────────┴──────────┘
            │
            ▼
       To encode Fold 1:
       1. Compute category means & Bühlmann Z using strictly Folds 2, 3, 4, 5
       2. Apply credibility formula: S_c = Z·ȳ_c + (1-Z)·ȳ_global
       3. Map values onto Fold 1 observations
       4. Observation i never sees its own y_i in its feature value!

1. Partition the training dataset into $M$ disjoint folds (e.g., $M = 5$ or $10$).
2. For each fold $m \in {1, 2, \dots, M}$:
   • Exclude fold $m$.
   • Compute category sample means $\bar{y}{c}^{(-m)}$, exposures $n_c^{(-m)}$, and the grand mean $\bar{y}{\text{global}}^{(-m)}$ strictly from the remaining $M - 1$ folds.
   • Calculate credibility-weighted values $\hat{S}_c^{\text{cred}, (-m)}$ using only out-of-fold data.
   • Map these values onto the observations in fold $m$.
3. For scoring prospective validation/test data or production policies, compute the credibility-weighted target encodings using the entire training dataset.

Additive Smoothing & Jitter

To provide an extra layer of regularization in machine learning trees, practitioners inject small Gaussian noise (jitter) into the encoded feature during training: $\tilde{S}_c = \hat{S}c^{\text{cred}} + \varepsilon$, where $\varepsilon \sim \mathcal{N}(0, \sigma^2{\text{noise}})$. This prevents decision trees from performing exact threshold splits on discrete credibility values.

Choosing the Base Level

Every categorical variable in a GLM is fitted relative to a base level, whose relativity is fixed at 1.000 and whose effect is absorbed into the intercept. The choice does not change the model's predictions, but it changes every reported coefficient and every standard error, so it is a presentation decision with real consequences.

Choose the base level with the largest exposure. Two reasons:

1. Standard errors are measured against the base. Each non-base coefficient's standard error reflects uncertainty in both that level and the base. A base level with thin data inflates the standard errors of every other level, making the whole variable look insignificant.
2. Readers anchor on the base. A relativity table whose 1.000 sits on a level containing 2% of exposure gives the reader no intuition for what "average" means.

Grouping Levels Before Fitting

A GLM assigns full credibility to every level of a categorical variable. The estimate for a territory with 8 claims is simply that territory's own experience, with no shrinkage toward the mean. The model warns you — the standard error will be large and the p-value high — but it does nothing about it.

That makes grouping a modelling decision the actuary must make, not one the software makes:

• Group by similarity of indicated effect, not alphabetically or geographically, and check that the grouping holds up out of sample.
• Set an exposure or claim-count floor below which a level cannot carry its own coefficient. State the floor in the report.
• Use a residual "other" level for the genuinely small remainder, and check that it is not absorbing levels with materially different experience.
• Re-check after grouping: deviance should barely rise while the number of parameters falls sharply.

Comparing the Options

For a PCPA project, grouping plus a clearly stated exposure floor is usually the right answer: it is defensible in a sentence, it satisfies the rubric criterion on variable treatment, and it avoids the leakage traps that target encoding introduces.