3.2 Missing Data Mechanisms & Imputation Strategies

1.1 Missing Data Mechanisms & Imputation Strategies

Exam Focus: For the CAS PCPA exam, missing data is not merely a technical annoyance to be eliminated through listwise deletion. Actuarial predictive models require strict adherence to regulatory standards, preservation of risk variance, and understanding of how missingness itself serves as a behavioral signal. Candidates must master Rubin's missing data taxonomy, diagnostic statistical tests, multiple imputation algorithms, and the actuarial mechanics of missing value indicator flags.

In property and casualty (P&C) ratemaking and underwriting modeling, missing values appear across virtually every data source—from motor vehicle records (MVRs) and credit-based insurance scores in personal auto to annual payroll audits and building construction specifications in commercial lines. How an actuary handles missing values directly influences the stability, profitability, and regulatory defensibility of the final rating algorithm.

---

1. Rubin's Missing Data Taxonomy in P&C Insurance

Donald Rubin (1976) established the foundational statistical framework classifying missing data mechanisms into three distinct categories based on the relationship between the probability of missingness and the underlying values. Let Y = (Y_obs, Y_mis) denote the complete data vector partitioned into observed components Y_obs and missing components Y_mis, and let M denote a binary missingness indicator matrix where M_i = 1 if record i is missing and M_i = 0 if observed.

                    ┌─────────────────────────────────────────────────────────┐
                    │               Rubin's Missingness Taxonomy              │
                    └─────────────────────────────────────────────────────────┘
                                                 │
         ┌───────────────────────────────────────┼───────────────────────────────────────┐
         ▼                                       ▼                                       ▼
  ┌──────────────┐                        ┌──────────────┐                        ┌──────────────┐
  │     MCAR     │                        │     MAR      │                        │     MNAR     │
  │   Missing    │                        │   Missing    │                        │   Missing    │
  │Completely at │                        │  at Random   │                        │Not at Random │
  │    Random    │                        │              │                        │              │
  └──────┬───────┘                        └──────┬───────┘                        └──────┬───────┘
         │                                       │                                       │
         ▼                                       ▼                                       ▼
  P(M|Yobs, Ymis)                         P(M|Yobs, Ymis)                         P(M|Yobs, Ymis)
     = P(M)                                  = P(M|Yobs)                             ≠ P(M|Yobs)
         │                                       │                                       │
         ▼                                       ▼                                       ▼
Complete-case analysis                  Complete-case analysis                  Standard imputation
remains unbiased, but                   introduces sample bias;                 fails; requires
loses statistical power.                imputation using Yobs                   pattern-mixture or
Rare in insurance data.                 restores unbiasedness.                  explicit missing flags.

Missing Completely at Random (MCAR)

Under MCAR, the event that data is missing is entirely independent of both observed covariates and unobserved missing values:

P(M = 1 | Y_obs, Y_mis) = P(M = 1)

• P&C Scenario: In a personal auto telematics program, an onboard diagnostics (OBD-II) dongle or smartphone app drops GPS coordinates for 30 seconds due to a random cellular tower handoff or transient satellite blackout. The loss of data has no correlation with the driver's aggressiveness, vehicle speed, weather, or location.
• Actuarial Implications: Complete-case analysis (listwise deletion) yields unbiased parameter estimates in Generalized Linear Models (GLMs). However, deleting records discards valuable statistical power and reduces earned exposure volume, increasing standard errors.

Missing at Random (MAR)

Under MAR, the propensity for a value to be missing depends systematically on observed data, but conditional on those observed variables, missingness is independent of the missing values themselves:

P(M = 1 | Y_obs, Y_mis) = P(M = 1 | Y_obs)

• P&C Scenario: In commercial property underwriting, building square footage is missing disproportionately for older structures built prior to municipal electronic recordkeeping in 1980. However, conditional on knowing the year of construction, municipal zoning classification, and construction type (all observed), the probability of square footage being missing does not depend on whether the building is exceptionally small or exceptionally massive.
• Actuarial Implications: Deleting incomplete records under MAR introduces severe selection bias. For example, dropping older policies skews the baseline rate toward newer construction. Imputation algorithms (such as MICE or predictive mean matching) conditioned on observed covariates can successfully restore unbiased parameter estimates.

Missing Not at Random (MNAR)

Under MNAR (non-ignorable missingness), the probability of missingness depends directly on the unobserved value itself, even after controlling for all available observed covariates:

P(M = 1 | Y_obs, Y_mis) ≠ P(M = 1 | Y_obs)

• P&C Scenario: In personal auto, applicants with poor driving records or prior undisclosed suspensions deliberately refuse to report their prior insurance carrier or length of prior coverage. Similarly, commercial general liability (CGL) contractors engaged in high-hazard subcontracting work omit reporting subcontractor payroll on renewal questionnaires to avoid premium surcharges.
• Actuarial Implications: Standard imputation models conditioned on observed data will produce biased estimates because the true missing values systematically occupy the high-risk tail. In actuarial ratemaking, MNAR cannot be resolved by standard statistical imputation; instead, actuaries preserve the signal by assigning missing records to a dedicated risk category or applying an explicit missing indicator flag.

---

2. Statistical Diagnostics for Missingness Mechanisms

Actuaries must empirically test and defend their missing data assumptions before selecting a remediation strategy.

Little's MCAR Test

Little (1988) proposed a global multivariate test for MCAR across all variables simultaneously. The dataset is partitioned into J distinct missingness patterns, where each pattern j has a unique subset of observed variables.

The test statistic evaluates whether the observed variable means within each missingness pattern differ significantly from the maximum likelihood estimates of the global means:

d² = ∑ [n_j · (ȳ_obs,j - μ̂_obs,j)ᵀ · (Σ̂_obs,j)⁻¹ · (ȳ_obs,j - μ̂_obs,j)]

Where:

• n_j is the number of observations exhibiting missingness pattern j.
• ȳ_obs,j is the vector of sample means for the observed variables in pattern j.
• μ̂_obs,j is the maximum likelihood estimate of the population mean vector for the observed variables in pattern j, estimated across the entire dataset via the Expectation-Maximization (EM) algorithm.
• Σ̂_obs,j is the estimated covariance matrix restricted to the variables observed in pattern j.

Under the null hypothesis that data is MCAR, d² asymptotically follows a chi-squared distribution:

d² ~ χ²(∑ p_j - p)

Where p_j is the number of observed variables in pattern j, and p is the total number of variables. If the p-value is less than 0.05, the null hypothesis is rejected, confirming the data is not MCAR (indicating either MAR or MNAR).

[!WARNING] Actuarial Exam Trap: In massive P&C policy databases (N > 100,000), Little's test has extreme statistical power. Even trivial, practically negligible deviations from MCAR will produce a statistically significant p-value (p < 0.001). Actuaries should complement Little's test with effect-size diagnostics and missingness indicator logistic regressions.

Missingness Indicator Logistic Regression

To diagnose whether a specific predictor X_k follows MCAR or MAR, create a binary indicator variable R_ik:

R_ik = 1 if X_ik is missing, and 0 if X_ik is observed.

Fit a logistic regression model predicting R_ik using all other observed covariates Z_i:

logit(P(R_ik = 1 | Z_i)) = α₀ + γᵀ Z_i

• If the Wald test or Likelihood Ratio Test indicates that γ = 0, the missingness of X_k does not depend on observed covariates, supporting the MCAR hypothesis.
• If any coefficients γ_j are statistically significant (e.g., driver age or vehicle garaging territory significantly predicts missing annual mileage), the missingness is MAR with respect to those covariates.

---

3. Imputation Techniques: Mechanics and Limitations

                                  ┌───────────────────────────────────┐
                                  │   Imputation Methods Comparison   │
                                  └───────────────────────────────────┘
                                                    │
         ┌──────────────────────────────────────────┼──────────────────────────────────────────┐
         ▼                                          ▼                                          ▼
  ┌──────────────┐                           ┌──────────────┐                           ┌──────────────┐
  │ Mean/Median  │                           │ KNN / Dist.  │                           │  MICE / PMM  │
  └──────┬───────┘                           └──────┬───────┘                           └──────┬───────┘
         │                                          │                                          │
         ▼                                          ▼                                          ▼
• Simple & fast                            • Multivariate local fit                   • Preserves uncertainty
• Crushes variance                         • Distance-metric sensitive                • PMM guarantees real donor
• Distorts covariance                      • O(N·D) computationally slow              • Rubin's rules pool results
• Unacceptable for tails                   • Unsuitable for large N                   • Industry gold standard

Mean, Median, and Mode Imputation

Single-value central tendency imputation replaces all missing values with a fixed summary statistic calculated from observed records:

• Continuous Features: Sample mean or sample median (preferred for skewed insurance quantities like policyholder income or building square footage).
• Categorical Features: Sample mode.

Mathematical Pitfalls in Actuarial Science:

1. Variance Deflation: Replacing missing entries with a single constant artificially reduces the sample variance. If a fraction p of records are missing and imputed with the sample mean x̄, the variance of the imputed feature X_imp shrinks to: Var(X_imp) = (1 - p) · Var(X_obs) This artificially deflates standard errors, causing statistical hypothesis tests (e.g., t-tests and GLM Wald tests) to generate false positives (Type I errors).
2. Covariance Attenuation: Imputing a constant weakens the correlation between the imputed predictor and the response variable (claim frequency or severity): Cov(X_imp, Y) = (1 - p) · Cov(X_obs, Y_obs)
3. Distributional Spikes: Mean or median imputation creates an artificial probability mass spike at the imputed value, distorting tree-based splitting criteria and kernel density estimations.

K-Nearest Neighbors (KNN) Imputation

KNN imputation replaces missing values in target record i with the weighted average (for continuous variables) or mode (for categorical variables) of the k most similar complete records in feature space.

• Distance Metric: For mixed data types (continuous vehicle age, binary homeowner status, categorical territory), Gower's distance is standard: S_ij = (∑ w_m · s_ijm) / (∑ w_m) Where s_ijm = 1 - (|x_im - x_jm| / R_m) for continuous variables with range R_m, and s_ijm ∈ {0, 1} for categorical variables.
• Limitations: KNN has computational complexity O(N · D) per missing record, making it computationally prohibitive for enterprise policy datasets containing millions of earned vehicle-years. Furthermore, KNN requires strict feature scaling; unscaled variables with large ranges (e.g., policy limits or building values) will dominate the neighbor search.

Multiple Imputation by Chained Equations (MICE) & Fully Conditional Specification (FCS)

MICE handles multivariate missing data by iteratively specifying separate univariate regression models for each variable conditioned on all other variables in the dataset.

The MICE Algorithm:

1. Initialize missing values with temporary random draws or median imputation.
2. For each variable Y_j with missingness (j = 1, ..., p):
   • Retract the current imputed values for Y_j, setting them back to missing.
   • Regress Y_j,obs on all other variables Y_-j using an appropriate generalized model:
     • Linear regression for normal continuous variables
     • Logistic regression for binary variables (e.g., prior accident flag)
     • Multinomial or ordinal logistic regression for categorical variables
     • Poisson or negative binomial for count variables
   • Draw new parameter values from their posterior predictive distribution: θ_j* ~ P(θ_j | Y_j,obs, Y_-j).
   • Impute missing entries Y_j,mis by drawing from the predictive distribution P(Y_j,mis | Y_-j, θ_j*).
3. Repeat step 2 across all variables for T cycles (typically 10 to 20 iterations) until the joint distribution stabilizes.
4. Repeat the entire process to generate m independent imputed complete datasets (typically m = 5 to 10).

                                 ┌──────────────────────────────────────┐
                                 │         MICE Iteration Cycle         │
                                 └──────────────────────────────────────┘
                                                     │
         ┌───────────────────────────────────────────┼───────────────────────────────────────────┐
         ▼                                           ▼                                           ▼
  ┌──────────────┐                            ┌──────────────┐                            ┌──────────────┐
  │ Continuous Y1│                            │   Binary Y2  │                            │    Count Y3  │
  │ (e.g. TIV)   │                            │ (e.g. Prior) │                            │ (e.g. Claims)│
  └──────┬───────┘                            └──────┬───────┘                            └──────┬───────┘
         │                                           │                                           │
         ▼                                           ▼                                           ▼
  Fit Linear Reg:                             Fit Logistic:                               Fit Poisson:
  Y1 ~ Y2 + Y3                                Y2 ~ Y1 + Y3                                Y3 ~ Y1 + Y2
         │                                           │                                           │
         ▼                                           ▼                                           ▼
Draw posterior θ1*                           Draw posterior θ2*                           Draw posterior θ3*
Impute missing Y1                            Impute missing Y2                            Impute missing Y3
         │                                           │                                           │
         └───────────────────► Repeat for T cycles ──┴───────────────────────────────────────────┘

Rubin's Rules for Pooling Parameter Estimates:

When fitting an actuarial pricing model to the m imputed datasets, the point estimates Q̂_l and variances Û_l from each dataset l ∈ {1, ..., m} are combined to incorporate both within-imputation and between-imputation uncertainty.

1. Pooled Point Estimate: Q̄ = (1 / m) ∑ Q̂_l
2. Within-Imputation Variance: Ū = (1 / m) ∑ Û_l
3. Between-Imputation Variance: B = [1 / (m - 1)] ∑ (Q̂_l - Q̄)²
4. Total Variance: T = Ū + (1 + 1 / m) · B

The term (1 + 1 / m) · B directly accounts for the extra variance introduced by having to guess the missing values, ensuring that final actuarial rate relativities reflect parameter uncertainty.

Predictive Mean Matching (PMM)

Predictive Mean Matching is a semi-parametric variant of MICE widely considered the gold standard for continuous insurance features with non-normal distributions.

PMM Algorithm:

1. Fit a linear regression model of observed values Y_obs on covariates X_obs, yielding estimated coefficients β̂ and residual variance σ̂².
2. Draw a random parameter set β* from the posterior distribution P(β | Y_obs, X_obs).
3. Calculate predicted values for all observed records: ŷ_i = X_i,obs · β*.
4. Calculate predicted values for all missing records: ŷ_j = X_j,mis · β*.
5. For each missing record j, identify donor records from the observed pool whose predictions ŷ_i are closest to ŷ_j based on absolute difference |ŷ_j - ŷ_i|.
6. Randomly select one donor from the top k closest candidates (typically k = 5) and assign that donor's actual observed value y_i,donor to the missing case.

Why PMM Excels in Actuarial Applications:

• Preserves Real Data Bounds: Imputed values are drawn exclusively from observed donor values. It is mathematically impossible for PMM to impute a negative claim count, negative building replacement cost, or fractional vehicle safety symbol.
• Maintains Non-linear Relationships: Because donor values reflect true underlying records, multi-modal distributions, discrete clustering, and heteroscedasticity are preserved without requiring rigid parametric distributional assumptions.

---

4. The Actuarial Missing Indicator Approach in Rating Algorithms

In competitive insurance markets, missingness is rarely an ignorable statistical defect. Rather, the presence of missing data often reflects consumer self-selection or underwriting friction. For example, an applicant who refuses to consent to a credit check or cannot produce proof of prior continuous coverage is demonstrably higher risk than an applicant with complete documentation.

The Dual-Variable GLM Formulation

To preserve the predictive signal of missingness in Generalized Linear Models without discarding records, actuaries construct a two-part representation for any continuous predictor X exhibiting missingness:

1. Imputed Continuous Variable (X):* X_i* = X_i if X_i is observed, and c if X_i is missing. Where c is an arbitrary fixed constant (conventionally chosen as 0, the sample median, or the sample mean).
2. Binary Missingness Indicator (I_miss): I_i,miss = 0 if X_i is observed, and 1 if X_i is missing.

Both variables are entered simultaneously into the GLM rating structure with a log link function:

ln(μ_i) = β₀ + β₁ · X_i* + β_miss · I_i,miss + ∑ β_k · Z_ik

Mathematical Interpretation of Coefficients:

• For an observed policyholder (I_i,miss = 0): ln(μ_i) = β₀ + β₁ · X_i + ∑ β_k · Z_ik The slope coefficient β₁ reflects the pure marginal effect of variable X strictly evaluated across policyholders with known, verified information.
• For a missing policyholder (I_i,miss = 1): ln(μ_i) = β₀ + β₁ · c + β_miss + ∑ β_k · Z_ik = (β₀ + β₁ · c + β_miss) + ∑ β_k · Z_ik The model assigns a dedicated baseline shift (β₁ · c + β_miss) to unobserved risks.

[!NOTE] Arbitrary Constant Independence: The choice of imputation constant c does not alter the model's fitted predictions, deviance, or the continuous slope β₁. If c is changed to c', the optimizer simply adjusts β̂_miss to β̂_miss + β̂₁ · (c - c'), leaving the net linear predictor perfectly invariant.

---

5. Actuarial Traps & Regulatory Pitfalls

                    ┌────────────────────────────────────────────────────────┐
                    │          Actuarial Traps & Regulatory Pitfalls         │
                    └────────────────────────────────────────────────────────┘
                                                 │
         ┌───────────────────────────────────────┼───────────────────────────────────────┐
         ▼                                       ▼                                       ▼
  ┌──────────────┐                        ┌──────────────┐                        ┌──────────────┐
  │   Jensen's   │                        │   Variance   │                        │  Regulatory  │
  │  Inequality  │                        │  Deflation   │                        │ Restrictions │
  └──────┬───────┘                        └──────┬───────┘                        └──────┬───────┘
         │                                       │                                       │
         ▼                                       ▼                                       ▼
Mean imputation in                      Underestimating tail                    State bans on imputing
log-link GLMs causes                    losses leads to inadequate              protected classes (race,
E[exp(Xβ)] ≥ exp(E[Xβ]),                reinsurance layers and                  gender) or surcharging
systematically under-                   underpriced excess                      missing credit scores
predicting pure premium.                severity tiers.                         (NAIC Model Act).

Trap 1: Distorting Loss Cost Estimates via Jensen's Inequality

In P&C ratemaking, claim frequency and severity are almost universally modeled using log-link GLMs (g(μ) = ln(μ), hence μ = exp(Xβ)). By Jensen's Inequality, for any convex function f and non-degenerate random variable Z:

E[f(Z)] ≥ f(E[Z])

Because the exponential function is strictly convex:

E[exp(Xβ)] > exp(E[Xβ]) = exp(x̄ · β)

If an actuary replaces missing values with the sample mean x̄, the model evaluates the loss cost at exp(x̄ · β) rather than the true expected value E[exp(Xβ)]. This mathematical error causes the model to systematically underestimate the expected pure premium for risks with missing values whenever the underlying distribution is right-skewed.

Trap 2: Artificially Deflating Risk Variance in Severity Tails

Commercial property and general liability claims exhibit extreme heavy tails (often modeled via Pareto or Log-Normal distributions). Imputing central values (mean or median) collapses the variance of exposure metrics (e.g., Total Insured Value, square footage, annual revenue). When these collapsed features are fed into severity models, the predicted risk distribution severely underrepresents the probability of shock losses, leading to deficient capital allocations and inadequate reinsurance detachment pricing.

Trap 3: Imputing Before the Data Is Split

Any imputation rule learned from the data — a mean, a median, a KNN neighbourhood, a MICE chain — is a quantity estimated from the target-bearing records. Fitting it on the full data set and then splitting into training and holdout partitions lets information from the holdout flow into the training records. The holdout is no longer independent, and out-of-sample performance is overstated.

The correct order is always:

1. Split first, into training and holdout (and folds, if cross-validating).
2. Learn the imputation rule on the training partition only.
3. Apply that stored rule to the holdout partition and to any future record scored in production.

The same discipline applies to every data-dependent treatment: winsorising caps, banding cut-points, level-grouping schemes, and credibility-weighted encodings. Each must be derived inside the training fold and then applied outward, never re-estimated on the data being used to judge the model.

[!WARNING] This trap is almost invisible in results: the model converges, the diagnostics look normal, and only the holdout statistics are quietly optimistic. Document the order of operations in the technical report so a reviewer can see that imputation was fitted on training data alone.