6.4 Under-fitting, Over-fitting & Cross-Validation

7.1 Insurance Data Splitting & Cross-Validation

Exam Focus: In academic machine learning, random cross-validation is often treated as the gold standard for model evaluation. In property and casualty (P&C) actuarial practice, applying standard random splits to insurance experience data is a fatal modeling error. Because insurance data exhibits strong temporal correlation, macroeconomic loss cost inflation, weather catastrophe clustering, and hierarchical policyholder structures, naive data partitioning introduces catastrophic data leakage. Candidates must master the distinct roles of training, validation, and holdout test sets, how to design chronological out-of-time (OOT) splits that mirror prospective ratemaking, and how to execute stratified and grouped cluster sampling.

Predictive models in property and casualty insurance are developed to establish prospective rates, underwrite upcoming renewal cohorts, and optimize future marketing or claims triage. Evaluating a model on historical data requires simulating this forward-looking operational environment. If the validation architecture leaks future information or fails to preserve natural clustering, the model will produce over-optimistic performance metrics that collapse upon deployment.

---

1. The Three-Way Partitioning Architecture

A disciplined actuarial modeling pipeline partitions historical policy and claim records into three distinct, non-overlapping subsets: the Training Set, the Validation Set, and the Holdout Test Set.

                          HISTORICAL INSURANCE DATASET
                                       │
         ┌─────────────────────────────┴─────────────────────────────┐
         ▼                                                           ▼
┌───────────────────────────────────────────────┐     ┌─────────────────────────────┐
│         Development / Modeling Sample         │     │      Holdout Test Set       │
│                    (80%)                      │     │            (20%)            │
└───────────────────────┬───────────────────────┘     └──────────────┬──────────────┘
                        │                                            │
          ┌─────────────┴─────────────┐                              │
          ▼                           ▼                              │
   ┌──────────────┐            ┌──────────────┐                      │
   │ Training Set │            │Validation Set│                      │
   │    (60%)     │            │    (20%)     │                      │
   └──────┬───────┘            └──────┬───────┘                      │
          │                           │                              │
          ▼                           ▼                              ▼
   Fits model parameters       Tunes hyperparameters,         Evaluates final
   (GLM coefficients,          evaluates early stopping,      generalization, pricing
   tree node splits)           compares algorithms            impact, and rate filings

Partitioning Roles in Actuarial Workflows

Mathematical Formulation: Empirical Risk vs. Generalization Risk

Let $\mathcal{D} = {(\mathbf{x}i, y_i, w_i)}{i=1}^n$ represent an insurance dataset where $\mathbf{x}_i$ is the rating vector, $y_i$ is the observed response (claim count, severity, or loss cost), and $w_i$ is earned exposure. The empirical training risk under deviance loss $d(y_i, \mu_i)$ is:

$R_{\text{train}}(\boldsymbol{\theta}) = \frac{1}{\sum_{i \in \mathcal{D}_{\text{train}}} w_i} \sum_{i \in \mathcal{D}_{\text{train}}} w_i d(y_i, g^{-1}(\mathbf{x}_i^T \boldsymbol{\theta}))$

Because optimization routines minimize $R_{\text{train}}(\boldsymbol{\theta})$ directly, $R_{\text{train}}$ systematically underestimates the true prospective loss cost discrepancy. The generalization error must be estimated on the independent holdout partition $\mathcal{D}_{\text{test}}$:

$R_{\text{gen}} \approx R_{\text{test}}(\hat{\boldsymbol{\theta}}) = \frac{1}{\sum_{i \in \mathcal{D}_{\text{test}}} w_i} \sum_{i \in \mathcal{D}_{\text{test}}} w_i d(y_i, g^{-1}(\mathbf{x}_i^T \hat{\boldsymbol{\theta}}))$

---

2. Temporal Validation & Out-of-Time (OOT) Splits

In standard cross-validation, observations are shuffled randomly across $k$ folds. In property and casualty insurance, random cross-validation causes severe, invalidating data leakage.

Why Random Splitting Fails in P&C Insurance

                              HAZARDS OF RANDOM SPLITTING IN P&C
                                               │
         ┌────────────────────────┬────────────┴────────────┬────────────────────────┐
         ▼                        ▼                         ▼                        ▼
  ┌──────────────┐         ┌──────────────┐          ┌──────────────┐         ┌──────────────┐
  │ Macro Trend  │         │ Weather Cat  │          │   Seasonal   │         │ Look-Ahead   │
  │  Inflation   │         │  Clustering  │          │   Patterns   │         │     Bias     │
  └──────┬───────┘         └──────┬───────┘          └──────┬───────┘          └──────┬───────┘
         │                        │                         │                        │
         ▼                        ▼                         ▼                        ▼
Severe medical/repair    Hail storm/freeze event    Winter ice or hurricane   Model trains on future
inflation in 2024 leaks  in June 2023 affects 500   driving season claims     rates/losses to predict
into 2021 predictions    policies in train & test   leak across random folds  past policy outcomes

1. Macroeconomic Trends and Social Inflation: Loss costs are subject to persistent, non-stationary inflation. Supply-chain disruptions, medical cost increases, vehicle repair technology changes (e.g., ADAS sensors in bumpers), and litigation trends (social inflation) cause loss costs to drift over time. Random splitting interweaves 2021 and 2024 claims, allowing the model to learn 2024 price levels when predicting 2021 policies—an impossible condition in prospective ratemaking.
2. Weather Catastrophe Clustering: P&C claim occurrences are not independent. An atmospheric river, hurricane, or severe convective hail storm creates thousands of correlated property and auto physical damage claims across a single 48-hour window. If policies exposed to the same storm are randomly scattered between training and test folds, the model memorizes the specific geographical footprint of that historical storm rather than learning true underlying risk characteristics.
3. Seasonal Variations: Claim frequency varies dramatically by month (e.g., winter collision peaks in northern states, summer recreational driving surges, autumnal hurricane activity). Random splits homogenize seasonal shocks across folds, masking the model's inability to handle cyclical claim spikes.
4. Look-Ahead Bias: In production, an insurer prices policies written today that will incur losses over the upcoming 12 to 24 months. Random splitting utilizes future data to predict the past, directly violating the temporal direction of insurance operations.

Chronological Out-of-Time (OOT) Architecture

To faithfully replicate prospective pricing, actuaries structure partitions strictly along chronological boundaries using Policy Year or Accident Year:

                           CHRONOLOGICAL OUT-OF-TIME SPLIT
  
  Policy Year:    2021            2022            2023            2024            2025
               ┌───────────────┬───────────────┬───────────────┬───────────────┬───────────────┐
               │               TRAINING SET                    │  VALIDATION   │    HOLDOUT    │
               │       (Policy Years 2021 - 2023)              │  (P.Y. 2024)  │  (P.Y. 2025)  │
               └───────────────────────────────────────────────┴───────────────┴───────────────┘
                                                                       │               │
                                                                       ▼               ▼
                                                                  Tuning &        Prospective
                                                                  Pruning         Evaluation

Actuarial Experience Alignment: Policy Year vs. Accident Year

When designing chronological splits, actuaries must respect the operational grain of the data:

• Policy Year (Underwriting Year): Groups all exposures and incurred losses arising from policies written within a specific calendar year. A policy written on December 31, 2023, remains in force through December 30, 2024. Policy year splits align perfectly with rating algorithm updates, as entire contracts and their complete loss experience are assigned to a single cohort.
• Accident Year (Occurrence Year): Groups all claims occurring in a calendar year, regardless of when the policy was written. While accident year data aligns with macroeconomic inflation and catastrophe events, matching exposures requires earned exposure unbundling.

[!IMPORTANT] Claim Development Lag (IBNR / IBNER): When selecting recent periods for validation or test sets (e.g., Policy Year 2024 or 2025), actuaries must recognize that recent experience is immature. Open claims will experience reserve development (IBNER), and late-reported claims have not yet emerged (IBNR). Actuaries must either develop losses to an ultimate basis using actuarial loss development factors (LDFs) before model evaluation or truncate validation windows to mature historical periods.

Rolling-Origin (Expanding Window) Cross-Validation

Rather than relying on a single temporal split, actuaries frequently evaluate model stability across multiple time periods using Rolling-Origin (Time-Series) Cross-Validation:

Fold 1:  [ Train: 2019 - 2021 ] ──► [ Validate: 2022 ]
Fold 2:  [ Train: 2019 - 2022 ] ──────► [ Validate: 2023 ]
Fold 3:  [ Train: 2019 - 2023 ] ──────────► [ Validate: 2024 ]

This structure ensures that training data always precedes validation data, testing whether the model's predictive superiority remains consistent across changing underwriting cycles and inflation regimes.

---

3. Stratified Splitting for Rare, High-Severity Events

Insurance loss data is characterized by extreme right-skewness and excess zeros: in personal auto, 90%–95% of policies incur zero claims; in commercial general liability, less than 2% of accounts incur losses, but a single environmental liability or construction defect claim can exceed $1,000,000.

                           THE IMBALANCED LOSS SPECTRUM
  
  Frequency:   ███████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░
               93% Zero Claims                       6.9% Small Claims   0.1% Shock
  
  Loss Dollars:░░░░░░░░░░░░░░░░░░░░░░░░░░░░           █████████████████████████████
               Zero Losses (0%)                      Attritional (40%)   Shock Losses (60%)

Why Unstratified Splitting Distorts Model Evaluation

If an actuary performs a naive unstratified partition on a commercial auto dataset with 10 shock claims exceeding $500,000:

• Pure random chance may assign 9 shock claims to the training set and only 1 to the test set.
• The training set severity will appear artificially inflated, while test set deviance, average loss cost, and lift metrics will be severely distorted.
• The actuary cannot determine whether poor test performance is caused by model failure or severe sampling imbalance.

Implementation of Multi-Tier Actuarial Stratification

To ensure statistical comparability across partitions, actuaries apply stratified sampling, preserving identical empirical distributions of critical risk attributes across all folds:

1. Binary Claim Occurrence Stratification: For frequency models, partition policies so that the proportion of claim-free policies ($N_i = 0$) versus claim-incurring policies ($N_i \ge 1$) is identical across training, validation, and test sets.
2. Severity Tier Stratification: For pure premium and severity models, partition claims across discrete loss layers:
   • Tier 1 (Attritional Losses): Claims with $0 < y_i \le 50,000$
   • Tier 2 (Large Losses): Claims with $50,000 < y_i \le 250,000$
   • Tier 3 (Shock / Catastrophe Losses): Claims with $y_i > 250,000$
3. Line of Business / Coverage Stratification: In multi-peril products (such as Commercial Multi-Peril or Homeowners), stratify by primary coverage (e.g., Property vs. Premises Liability vs. Water Backup) to guarantee that each partition contains representative coverage mixes.

# Conceptual Actuarial Stratification Variable Construction
# Combines coverage line and loss severity tier into a single composite stratum
def build_actuarial_stratum(policy):
    if policy.claim_count == 0:
        return f"{policy.line}_zero"
    elif policy.total_loss < 50000:
        return f"{policy.line}_attritional"
    elif policy.total_loss < 250000:
        return f"{policy.line}_large"
    else:
        return f"{policy.line}_shock"

---

4. Grouped / Cluster Splitting for Hierarchical Entities

Many property and casualty portfolios contain multiple observations linked to the same underlying insured entity. Examples include:

• Personal Auto Policies with Multiple Vehicles: A single household policy covering three vehicles and four drivers.
• Longitudinal Policy Renewals: A commercial enterprise insured with the carrier across six consecutive annual policy terms (2018–2023).
• Commercial Fleet Accounts: A trucking firm operating 250 power units, each represented as a distinct vehicle-year record.
• Commercial Property Multi-Location Schedules: A retailer with 40 individual store locations insured under a single commercial property schedule.

                         HIERARCHICAL ENTITY STRUCTURE
                                       │
                        Commercial Account: Fleet Corp
                        (Account ID: ACC-948102)
                                       │
         ┌─────────────────────────────┼─────────────────────────────┐
         ▼                             ▼                             ▼
    Vehicle 101                   Vehicle 102                   Vehicle 103
  (Long-Haul Semi)             (Delivery Box Truck)           (Service Van)
         │                             │                             │
  Policy Year 2022              Policy Year 2022              Policy Year 2022
  Policy Year 2023              Policy Year 2023              Policy Year 2023
  Policy Year 2024              Policy Year 2024              Policy Year 2024
  
  MANDATE: All 9 records MUST be assigned to the same partition!

The Danger of Intra-Cluster Correlation Leakage

Individual vehicles or locations within an account share powerful unobserved risk characteristics:

• The commercial fleet shares maintenance standards, driver safety protocols, geographic terminal locations, and telematics speed governances.
• A personal auto household shares garaging security, neighbourhood road hazards, and shared vehicle operation between family members.
• Multi-year policy renewals reflect invariant policyholder temperament, credit behavior, and property upkeep.

If records from the same entity are split randomly across training and validation sets:

• The model learns the idiosyncratic risk profile of that specific household or account in the training set (e.g., memorizing specific high-hazard driver combinations or vehicle fleet age patterns).
• When predicting the validation records for the same household, the model appears brilliantly accurate.
• In production, when the model is applied to entirely new accounts, predictive performance collapses because the model memorized customer identity rather than learning generalizable actuarial relationships.

Algorithmic Grouping Rule

Under Grouped / Cluster Splitting, the sampling unit is the primary entity identifier ($ID_{\text{entity}}$), rather than the individual policy or exposure row. All records sharing the same grouping key must reside exclusively within the training set, the validation set, or the holdout test set:

$\mathcal{S}_{\text{train}} \cap \mathcal{S}_{\text{val}} = \emptyset, \quad \mathcal{S}_{\text{train}} \cap \mathcal{S}_{\text{test}} = \emptyset, \quad \mathcal{S}_{\text{val}} \cap \mathcal{S}_{\text{test}} = \emptyset$

Where $\mathcal{S}k = {ID{\text{entity}} \in \text{Partition}_k}$.

---

5. Actuarial Traps & Exam Pitfalls

                    ┌────────────────────────────────────────────────────────┐
                    │          Actuarial Traps in Data Splitting             │
                    └────────────────────────────────────────────────────────┘
                                                 │
         ┌───────────────────────────────────────┼───────────────────────────────────────┐
         ▼                                       ▼                                       ▼
  ┌──────────────┐                        ┌──────────────┐                        ┌──────────────┐
  │ Random K-Fold│                        │ Immature OOT │                        │ Account-Level│
  │ on Time-Data │                        │ Validation   │                        │ Record Split │
  └──────┬───────┘                        └──────┬───────┘                        └──────┬───────┘
         │                                       │                                       │
         ▼                                       ▼                                       ▼
Leaks catastrophe weather               Recent policy years have                Splitting renewals or
events and inflation into               substantial IBNR/IBNER;                 multi-vehicle policies
past periods; results                   unadjusted losses under-                allows model to memorize
in production failure.                  state actual true risk.                 customer identity.

Trap 1: Submitting Rate Filings Based on In-Sample or Random Cross-Validation Metrics

Insurance department regulators and actuarial peer reviewers will challenge any filing where model separation diagnostics (such as Gini coefficients or lift charts) are calculated in-sample or using naive random k-fold CV on multi-year data. Regulators recognize that random CV masks the effects of loss inflation and catastrophe clustering, overstating rate adequacy.

Trap 2: Neglecting Loss Development on Recent Out-of-Time Partitions

When validating on the most recent completed policy year (e.g., using 2024 data evaluated in early 2025), reported losses are immature. If the actuary calculates validation deviance on raw reported losses, low-frequency, long-tail liability claims will appear absent. The model will appear to severely over-predict losses, leading the actuary to improperly deflate base rates. Actuaries must apply policy-year loss development factors or restrict OOT validation to mature years.

Trap 3: Splitting Multi-Car Policies Across Folds

A candidate builds an auto frequency model. Policies covering two cars (e.g., Mom and Teenager) have their vehicle records split: Car 1 goes to train, Car 2 goes to test. Because Car 1 and Car 2 share the same household address, territory, and policyholder credit score, the model achieves near-perfect validation accuracy by recognizing household commonalities. In production, single-car policies and new customers perform poorly.