7.2 Simple Quantile Plots & Measuring Lift

7.2 Lift Charts & Quantile Relativity Plots

Exam Focus: In standard regression analysis, models are evaluated using global goodness-of-fit metrics such as $R^2$, Mean Squared Error (MSE), or Akaike Information Criterion (AIC). In property and casualty ratemaking, however, the primary business objective is risk segmentation—the ability to accurately sort insureds from lowest risk to highest risk so that rates reflect true expected loss costs. The Single Lift Chart (also known as a Quantile Plot, Loss Cost Lift Chart, or Pure Premium Relativity Plot) is the fundamental actuarial diagnostic for demonstrating separation power, calibration, and rank-ordering. Candidates must master its step-by-step construction, the mandatory role of exposure weighting, the top-to-bottom decile ratio, and how to detect overfitting.

An insurance pricing model can achieve a mathematically respectable deviance score while failing commercially. If a model cannot distinguish between high-risk and low-risk drivers, an insurer charging average rates will experience severe adverse selection: competitors with superior segmentation will poach the profitable preferred accounts and dump the hazardous risks onto the insurer's book. Lift charts provide the visual and quantitative proof of risk separation.

---

1. Step-by-Step Construction Methodology

Constructing a single lift chart requires a disciplined, four-step actuarial procedure performed on an independent holdout test dataset or out-of-time validation dataset.

                         LIFT CHART CONSTRUCTION WORKFLOW
  
  ┌────────────────────────────────────────────────────────────────────────┐
  │ Step 1: Predict Pure Premium / Loss Cost                               │
  │ For each holdout policy i, calculate y_hat_i (e.g., Freq x Sev)        │
  └───────────────────────────────────┬────────────────────────────────────┘
                                      │
                                      ▼
  ┌────────────────────────────────────────────────────────────────────────┐
  │ Step 2: Sort Policies in Ascending Order                               │
  │ Rank policies: y_hat_(1) <= y_hat_(2) <= ... <= y_hat_(n)              │
  └───────────────────────────────────┬────────────────────────────────────┘
                                      │
                                      ▼
  ┌────────────────────────────────────────────────────────────────────────┐
  │ Step 3: Divide into Quantiles of EQUAL EARNED EXPOSURE                 │
  │ Accumulate earned exposure w_i into 10 deciles (or 20 vigintiles)      │
  │ Each decile contains exactly 10% of total portfolio earned exposure    │
  └───────────────────────────────────┬────────────────────────────────────┘
                                      │
                                      ▼
  ┌────────────────────────────────────────────────────────────────────────┐
  │ Step 4: Compute Actual vs. Predicted Loss Cost per Quantile            │
  │ Plot average actual loss cost Y_bar_k against predicted L_hat_k       │
  └────────────────────────────────────────────────────────────────────────┘

Step 1: Predict Pure Premium (Loss Cost)

For every observation $i$ in the holdout partition, generate the model's predicted pure premium (expected loss cost per unit of exposure), denoted $\hat{L}_i$:

• In a direct pure premium model (such as a Tweedie compound Poisson-Gamma GLM or GBDT), $\hat{L}_i = \hat{\mu}_i$.
• In a two-part frequency-severity model, $\hat{L}_i = \hat{\lambda}_i \times \hat{S}_i$, where $\hat{\lambda}_i$ is the predicted claim frequency (claims per earned exposure unit) and $\hat{S}_i$ is the predicted conditional claim severity (cost per claim).

Step 2: Sort Policies Ascending by Predicted Loss Cost

Sort all holdout policies in strictly ascending order of their predicted loss cost:

$\hat{L}_{(1)} \le \hat{L}_{(2)} \le \hat{L}_{(3)} \le \dots \le \hat{L}_{(n)}$

Policies predicted to have the lowest expected loss cost (preferred risks) are placed first; policies predicted to have the highest expected loss cost (hazardous risks) are placed last.

Step 3: Partition into Quantiles of Equal Earned Exposure

This is the most critical actuarial distinction in lift chart construction. Quantiles must be formed based on equal earned exposure, NOT equal policy counts.

Let $w_i$ represent the earned exposure (e.g., earned car-years, payroll thousands, or home-years) of policy $i$. The total portfolio earned exposure is:

$W_{\text{total}} = \sum_{i=1}^n w_i$

For a 10-decile chart, each decile $\mathcal{Q}_k$ ($k \in {1, 2, \dots, 10}$) must contain exactly $10%$ of the total earned exposure:

$W_k = \sum_{i \in \mathcal{Q}_k} w_i = \frac{W_{\text{total}}}{10}$

Policies are accumulated into Decile 1 until $\sum w_i = 0.10 W_{\text{total}}$, then Decile 2 until $0.20 W_{\text{total}}$, and so on through Decile 10.

[!IMPORTANT] Why Equal Policy Counts Cause Statistical Distortion: In personal auto, an insured who cancelled their policy after 18 days has an earned exposure of $w_i = 18 / 365 = 0.05$ car-years. A loyal policyholder with two annual renewals has $w_i = 1.0$ car-year. If quantiles are divided by raw policy count, short-term, partial-term, and endorsement policies carry the same weight as full-year policies. Because partial-term policies often cluster in volatile sub-segments (e.g., non-pay cancellations), equal policy count binning introduces severe sampling noise and breaks the statistical credibility of the loss cost calculation.

Step 4: Compute Actual and Predicted Metrics per Quantile

For each decile $k$, compute the exposure-weighted Average Predicted Loss Cost ($\bar{\hat{L}}_k$) and Average Actual Loss Cost ($\bar{Y}_k$):

$\bar{\hat{L}}_k = \frac{\sum_{i \in \mathcal{Q}_k} w_i \hat{L}_i}{\sum_{i \in \mathcal{Q}_k} w_i} = \frac{\sum_{i \in \mathcal{Q}_k} w_i \hat{L}_i}{W_k}$

$\bar{Y}_k = \frac{\sum_{i \in \mathcal{Q}_k} \text{Loss}_i}{\sum_{i \in \mathcal{Q}_k} w_i} = \frac{\sum_{i \in \mathcal{Q}_k} \text{Loss}_i}{W_k}$

Where $\text{Loss}_i$ is the actual incurred loss for policy $i$ in the holdout period.

To make the chart easily interpretable for executive stakeholders and regulators, actuaries often normalize both metrics by dividing by the overall portfolio average loss cost ($\bar{Y}_{\text{total}} = \sum \text{Loss}i / W{\text{total}}$), converting dollar values into Loss Cost Relativities:

$\text{Predicted Relativity}_k = \frac{\bar{\hat{L}}_k}{\bar{Y}_{\text{total}}}, \quad \text{Actual Relativity}_k = \frac{\bar{Y}_k}{\bar{Y}_{\text{total}}}$

---

2. Interpreting Lift Charts: Slope, Monotonicity, and Calibration

A lift chart displays two lines across the 10 deciles: the Predicted Loss Cost line (or relativity) and the Actual Loss Cost line.

  Loss Cost Relativity
      │
  3.0 │                                                      ▲ Actual Loss Cost
      │                                                ┌─────┘ (Tracks predicted)
  2.5 │                                          ┌─────┘
      │                                    ┌─────┘ ─── Predicted Loss Cost
  2.0 │                              ┌─────┘
      │                        ┌─────┘
  1.5 │                  ┌─────┘
      │            ┌─────┘
  1.0 │═════════════════════════════════════════════════════ Portfolio Average = 1.0
      │      ┌─────┘
  0.5 │┌─────┘
      │┘
  0.0 └────────────────────────────────────────────────────────►
       D1    D2    D3    D4    D5    D6    D7    D8    D9    D10
                    Deciles of Predicted Loss Cost (Equal Exposure)

Key Diagnostic Dimensions

1. Steepness of Slope (Separation Power): The steeper the upward progression from Decile 1 to Decile 10, the greater the model's ability to differentiate between low-risk and high-risk insureds. A steep curve indicates that the model has identified powerful predictive rating factors.
2. Monotonicity (Rank-Order Integrity): In an actuarially sound model, the actual loss costs must increase strictly monotonically from each decile to the next: $\bar{Y}_1 < \bar{Y}_2 < \bar{Y}_3 < \dots < \bar{Y}_{10}$ If Decile 5 has a higher actual loss cost than Decile 6, the model exhibits a non-monotonic inversion. While minor inversions in intermediate deciles can occur due to random sample variance in smaller datasets, widespread non-monotonicity proves the model is sorting on sample noise.
3. Calibration (Tracking Accuracy): At every decile, the Actual Loss Cost line should closely track the Predicted Loss Cost line. If the predicted curve is steep (predicting a 4.0x spread) but the actual line is much flatter (showing only a 2.0x spread), the model suffers from systematic over-dispersion and extreme relativity exaggeration.

Diagnostic Patterns in Single Lift Charts

         IDEAL LIFT                     FLAT (USELESS) LIFT              OVERFIT / NOISY LIFT
    Rel                          Rel                             Rel
    │                     ▲      │                               │        ▲
    │               ┌─────┘      │                               │       ╱ ╲    ▲
    │         ┌─────┘            │────────────────────────       │  ┌───┘   ╲  ╱ ╲
    │   ┌─────┘                  │                               │ ┌┘        ╲╱   └──
    └───┴─────────────────►      └───┴────────────────────►      └─┴─────────────────►
        D1             D10           D1                D10         D1             D10

---

3. Quantifying Lift: The Decile Relativity Ratio

While graphical lift charts provide intuitive visual diagnostics, actuaries require a scalar metric to benchmark competing models. The primary summary metric is the Top-to-Bottom Decile Lift Ratio ($L_{10/1}$):

$L_{10/1} = \frac{\bar{Y}_{10}}{\bar{Y}_1} = \frac{\text{Actual Loss Cost in Decile 10}}{\text{Actual Loss Cost in Decile 1}}$

Actuarial Interpretation of the Ratio

• A lift ratio of $L_{10/1} = 6.0$ indicates that the 10% of policies sorted into the highest predicted risk tier experience six times the loss cost per unit of exposure as the 10% sorted into the lowest risk tier.
• When comparing two models on the same holdout dataset, the model that achieves a higher, statistically credible, and monotonic $L_{10/1}$ ratio provides superior underwriting segmentation.

Empirical Benchmarks Across Property & Casualty Lines

---

4. In-Sample vs. Out-of-Sample Lift: Detecting Overfitting

The ultimate test of an actuarial model is whether its risk differentiation generalizes to unseen data. Comparing In-Sample (Training) Lift against Out-of-Sample (Holdout / Out-of-Time) Lift is the most effective diagnostic for uncovering over-parameterized models.

  Loss Cost Relativity
      │                                             In-Sample Training Lift (Overfit)
  5.0 │                                            . ───────────────────────────▲ (12.0x Lift)
      │                                           /                             │
  4.0 │                               ┌───────────┘                             │
      │                               │                                         │
  3.0 │                   ┌───────────┘             Out-of-Sample Holdout Lift   │
      │                   │                        . ─────────────────────▲     │
  2.0 │       ┌───────────┘             ┌──────────┘                      │     │ Severe Overfit
      │       │                   ┌─────┘                                 │     │ Shrinkage!
  1.0 │───────┴═══════════════════╧═══════════════════════════════════════╧═════╪═════════════
      │ ┌─────┘             ┌─────┘                                       │ 3.2x│
  0.0 └─┴───────────────────┴─────────────────────────────────────────────┴─────┴────────────►
        D1                  D4                  D7                       D10

The Mechanics of Lift Decay

When an over-parameterised model is fit to insurance data — for example a GLM in which every level of a 40-level territory variable and every level of an 18-level construction class carries its own coefficient — it memorizes idiosyncratic noise—such as a single policyholder with three minor speeding tickets who happened to suffer a catastrophic $400,000 bodily injury loss.

• On the Training Set: The algorithm creates high-order leaves isolating this specific customer profile, assigning them an astronomical predicted rate. The in-sample lift chart looks spectacular, with a massive Decile 10 loss cost and an in-sample ratio of $L_{10/1} = 12.5$.
• On the Holdout Set: Other policyholders with that same profile do not suffer shock losses. The out-of-sample Decile 10 actual loss cost collapses. The holdout lift ratio drops to $3.2$, and intermediate deciles become non-monotonic.

Actuarial Rule of Generalization Stability

A model is statistically robust if its out-of-sample lift ratio retains at least 80%–90% of its in-sample lift ratio while maintaining strict monotonicity. If holdout lift degrades by more than 30%, or if the outer deciles exhibit severe regression to the mean, the actuary must apply structural constraints, increase L2 regularization, or restrict tree depth.

---

5. Actuarial Traps & Exam Pitfalls

                    ┌────────────────────────────────────────────────────────┐
                    │          Actuarial Traps in Single Lift Charts         │
                    └────────────────────────────────────────────────────────┘
                                                 │
         ┌───────────────────────────────────────┼───────────────────────────────────────┐
         ▼                                       ▼                                       ▼
  ┌──────────────┐                        ┌──────────────┐                        ┌──────────────┐
  │ Unweighted   │                        │ Confusing    │                        │ In-Sample    │
  │ Policy Count │                        │ Lift with    │                        │ Validation   │
  │ Partitioning │                        │ Calibration  │                        │ Presentation │
  └──────┬───────┘                        └──────┬───────┘                        └──────┬───────┘
         │                                       │                                       │
         ▼                                       ▼                                       ▼
Short-term cancelled                    A model can have steep                  Presenting training lift
policies distort quantile               lift while under-pricing                creates false confidence;
loss costs; must bin                    the entire book by 20%;                 overfitting is exposed
by earned exposure.                     check base rate level.                  only on holdout data.

Trap 1: Partitioning Quantiles by Policy Count Instead of Earned Exposure

On the CAS PCPA exam, questions frequently test whether candidates know how quantiles are formed. Dividing data into 10 groups of equal row count is a standard data science practice, but in actuarial science it is an disqualifying error. An earned car-year represents the standardized unit of risk. Binning by policy count allows short-term policies (with minimal exposure and volatile loss ratios) to distort decile loss costs.

Trap 2: Confusing Strong Lift with Accurate Calibration

A model can exhibit a magnificent, perfectly monotonic 8.0x lift ratio while being completely miscalibrated. For example, if a model's predicted loss costs in every decile are exactly 20% lower than actual loss costs, the lift curve will look wonderfully steep and monotonic, but deploying those rates will cause an underwriting loss. Lift measures relative risk discrimination (ranking); overall actual-to-expected (A/E) ratios measure absolute rate level calibration.

Trap 3: Presenting Training Set Lift to Regulators

Submitting in-sample lift charts in an actuarial rate filing violates Actuarial Standard of Practice (ASOP) No. 56 (Modeling). The actuary must disclose the model's performance on independent, out-of-time validation data to demonstrate that filed rate differentials reflect genuine prospective risk separation rather than sample memorization.