7.1 Actual vs. Predicted, Calibration & Loss Ratio Charts

Calibration: The Complement to Discrimination

While discrimination metrics (such as the Gini coefficient and ROC AUC covered in Section 7.4) determine how effectively a model separates low-risk policies from high-risk policies, calibration metrics evaluate whether the predicted dollar amounts equal true expected loss costs.

In casualty actuarial practice, an uncalibrated model is dangerous:

• If rates are set according to an under-predicting model (Actual-to-Expected ratio $> 1.0$), collected premiums will be inadequate to cover incurred claims and expenses, resulting in underwriting losses, combined ratio deterioration, and regulatory solvency intervention.
• If rates are set according to an over-predicting model (Actual-to-Expected ratio $< 1.0$), premiums will be commercially uncompetitive, causing policyholder defection, declining renewal retention, and adverse selection.

Rigorous model validation requires testing calibration at two distinct levels: global calibration across the total portfolio and marginal calibration across individual rating dimensions.

---

Global Calibration: The Actual-to-Expected (A/E) Ratio

The most fundamental calibration metric is the portfolio-wide Actual-to-Expected (A/E) ratio (also termed the Actual-over-Expected loss ratio):

$\text{A/E}_{\text{global}} = \frac{\sum_{i=1}^n y_i}{\sum_{i=1}^n \hat{y}_i} = \frac{\sum_{i=1}^n y_i}{\sum_{i=1}^n e_i \hat{\mu}_i}$

Where:

• $y_i$ is the actual observed aggregate loss dollars incurred by policy $i$.
• $\hat{y}_i = e_i \hat{\mu}_i$ is the model-predicted aggregate loss dollars for policy $i$, calculated as earned exposure $e_i$ multiplied by the predicted pure premium $\hat{\mu}_i$.

Actuarial Interpretation of the Global A/E Ratio

• $\text{A/E} = 1.00$ (Perfect Aggregate Calibration): The total dollar volume of losses predicted by the model exactly matches the total dollar volume of losses observed in the validation experience. The portfolio base rate is properly centered.
• $\text{A/E} > 1.00$ (Systematic Under-prediction): Actual losses exceed predicted losses. The model underestimates overall risk costs. If rates are implemented without upward base rate adjustment, the insurer will experience an underwriting loss.
• $\text{A/E} < 1.00$ (Systematic Over-prediction): Predicted losses exceed actual losses. The model overestimates risk costs, which will lead to uncompetitive pricing in the marketplace.

The Fallacy of Relying Solely on Global Calibration

[!IMPORTANT] A global A/E ratio of exactly $1.00$ is a necessary but not sufficient condition for model validity. Aggregate balance can easily disguise catastrophic offsetting miscalibrations between rating classes.

Consider an auto bodily injury model evaluated on a holdout portfolio of $50,000$ car-years:

• Total Actual Losses: $$25,000,000$
• Total Predicted Losses: $$25,000,000$
• $\text{Global A/E} = \frac{$25,000,000}{$25,000,000} = 1.000$ (Appears flawless)

However, decomposing the portfolio into youthful vs. mature drivers reveals severe localized failure:

Because the youthful driver undercharge exactly balances the mature driver overcharge in total dollars, the global A/E ratio fails to detect the distortion. To uncover these structural defects, actuaries must analyze marginal calibration.

---

Marginal Calibration: One-Way Actual vs. Expected Plots

Marginal calibration diagnostics (commonly known in casualty ratemaking as one-way actual vs. expected plots or univariate lift graphs) evaluate model fidelity across discrete categories or continuous intervals of every candidate rating variable.

Mathematical Formulation

Let $X_j$ represent a rating variable (e.g., Driver Age, Vehicle Body Type, Territory, Home Age, Construction Class). We partition the domain of $X_j$ into $K$ distinct bins $b \in {1, 2, \dots, K}$. For continuous variables, bins are created using deciles of earned exposure or standardized actuarial age groupings.

For each bin $b$, the actuary computes three quantities on an independent holdout or out-of-time dataset:

1. Actual Loss Cost (Actual Pure Premium): $\text{ALC}_b = \frac{\sum_{i \in b} y_i}{\sum_{i \in b} e_i}$
2. Predicted Loss Cost (Expected Pure Premium): $\text{PLC}_b = \frac{\sum_{i \in b} e_i \hat{\mu}_i}{\sum_{i \in b} e_i}$
3. Marginal Segment A/E Ratio: $\text{A/E}_b = \frac{\text{ALC}_b}{\text{PLC}_b} = \frac{\sum_{i \in b} y_i}{\sum_{i \in b} e_i \hat{\mu}_i}$

Dual-Axis Visual Architecture

A standard actuarial one-way plot uses a dual-axis layout:

• Primary Left Y-Axis (Loss Cost Scale): Two overlaid lines tracking $\text{ALC}_b$ (solid line with observed markers) and $\text{PLC}_b$ (dashed model curve).
• Secondary Right Y-Axis (Exposure Volume): A vertical bar chart showing total earned exposure $e_b$ in each bin, providing visual credibility assessment.

                      ONE-WAY ACTUAL VS. EXPECTED: DRIVER AGE BINS

    Loss Cost ($)
    [Primary Axis]                                               Earned Exposure (Car-Years)
                                                                 [Secondary Axis, Bars]
    $1,200 ┼───────────────────────────────────────────────────┬ 25,000
           │  *                                                │
    $1,000 ┼───\───────────────────────────────────────────────┼ 20,000
           │    \*                                      *      │
      $800 ┼─────\*────────────────────────────────────╱───────┼ 15,000
           │      \*                                 *╱        │
      $600 ┼───────\──*                           *──╱─────────┼ 10,000
           │        \  \                         ╱  ╱   ██     │
      $400 ┼─────────\──\───────────────────────╱──╱────██─────┼ 5,000
           │   ██     \──*───*───*─────*───*───*──╱     ██     │
      $200 ┼───██─────██──██──██──██──██──██──██────────██─────┼ 0
           └───┴──────┴───┴───┴───┴───┴───┴───┴─────────┴─────┘
             16-20  21-25 26-30 31-40 41-50 51-60 61-70  71-80  81+
             [Low]   [--- Moderate to High Exposure ---]   [Low]

             Legend:
             * ── * : Actual Loss Cost (Observed)
             - - -  : Predicted Loss Cost (Model)
             █████  : Earned Exposure (Car-Years)

Diagnostic Patterns and Actuarial Remedies

1. Well-Calibrated Factor: The predicted line closely tracks the actual loss cost markers across all high-exposure bins, with segment A/E ratios hovering tightly between $0.95$ and $1.05$.
2. Missing Non-Linearity (Functional Misspecification): If the actual loss costs display a steep U-shaped curve across age bins (elevated losses for teen drivers, low losses for middle-aged drivers, rising losses for drivers aged 75+) while the model line is linear, the model has misspecified the predictor's functional form. Remedy: Replace the linear term with restricted cubic splines, fractional polynomials, or non-linear tree splits.
3. Unmodeled Interaction Effect: If an entire segment of a one-way plot (e.g., urban sports car drivers) exhibits actual loss costs substantially above predicted values, a multi-way interaction exists between territory and vehicle class that the additive linear predictor failed to capture. Remedy: Include an interaction term $\beta_{\text{terr} \times \text{veh}}$ or utilize gradient-boosted trees to detect higher-order feature splits.
4. Credibility-Driven Fluctuations: Large divergences in bins with minimal exposure (e.g., drivers aged 85+ representing $< 1%$ of total car-years) typically reflect sampling noise rather than model misspecification. Remedy: Apply actuarial credibility shrinkage (such as Bühlmann-Straub credibility) or collapse small edge bins.

Plots of Actual vs. Predicted

The simplest and most easily understood diagnostic for comparing competing models is a plot of the actual target on the y-axis against the predicted target on the x-axis. If a model fits well, actual and predicted follow each other closely and the points cluster around the 45-degree line. Comparing two such plots side by side usually settles which model fits better without any statistics at all.

The CAS Generalized Linear Models for Insurance Rating monograph attaches three cautions to this exhibit, and all three are examinable:

1. Build it on holdout data. On training data an over-fitted model produces a beautiful actual-versus-predicted plot and no predictive power on unseen risks. The plot is a validation tool only if the data is unseen.

2. Aggregate before plotting. With hundreds of thousands of records, a raw scatter is an unreadable cloud. The standard approach is to sort the data by the predicted target, group it into 100 buckets each holding the same aggregate model weight, and plot the average actual against the average predicted within each bucket.

3. Use a log scale. Loss data spans orders of magnitude, so on a linear scale a handful of very large values squashes everything else against the axes. Plotting both axes on a log scale makes the body of the distribution readable.

The fourth pattern is the one that matters most for a rating plan. A model whose predictions are too compressed still ranks risks correctly, so its lift and Gini look acceptable, but every relativity is too close to 1.0 and the plan leaves segmentation on the table.

Loss Ratio Charts

A loss ratio chart replaces pure premium with loss ratio, and it is the exhibit most likely to persuade a non-technical audience. Construction:

1. Sort the data by predicted loss ratio, defined as predicted loss cost divided by the premium currently charged.
2. Bucket into quantiles with equal exposure volume.
3. Within each bucket, compute the actual loss ratio.
4. Plot actual loss ratio by bucket.

The interpretation is direct. If the current rating plan were perfect, every bucket would show the same loss ratio. A chart where the buckets spread — say from 38% in the first decile to 72% in the tenth — is evidence that the model has found segments the current plan prices wrongly, and therefore that it out-performs that plan.

The monograph makes the practical case for this chart explicitly: its advantage over quantile plots and double lift charts is that it is simple to understand and explain. Loss ratio is the metric every underwriter, product manager and executive already uses, so the exhibit needs no translation.

[!WARNING] A loss ratio chart compares your model against the current rating plan, not against truth. A flat loss ratio chart means the model finds no mispricing the current plan has missed — which may mean the model is weak, or may mean the existing plan is already good. Read it alongside a quantile plot, which measures the model's own ability to separate risks.