4.3 Overdispersion, Quasi-Poisson & Negative Binomial

The Problem of Overdispersion in Insurance Data

In classical Poisson regression, the conditional variance is constrained to equal the conditional mean ($\text{Var}(Y \mid \mathbf{x}) = \mathbb{E}[Y \mid \mathbf{x}] = \mu$). In empirical property and casualty insurance datasets, this condition is almost never satisfied. Instead, actuaries encounter overdispersion:

$\text{Var}(Y \mid \mathbf{x}) > \mathbb{E}[Y \mid \mathbf{x}]$

Underdispersion ($\text{Var}(Y) < \mathbb{E}[Y]$) is virtually non-existent in general insurance, as loss occurrences are characterized by volatility, clustering, and latent risk factors.

---

Root Causes of Overdispersion in P&C Portfolios

Overdispersion does not imply that the data are corrupt; rather, it reflects genuine physical and behavioral characteristics of insurance risks:

1. Unobserved Policyholder Heterogeneity (Omitted Variable Bias)

No matter how granular an insurer's rating classification is, observable variables (such as driver age, territory, vehicle make, and credit score) capture only a fraction of true risk variance. Within any rating cell, unobservable traits—such as driver aggressiveness, reaction speed, daily route congestion, vehicle maintenance, and texting while driving—create substantial latent risk differences. This unmodeled variation causes claim counts to scatter far more widely than a homogeneous Poisson process predicts.

2. Claim Contagion and Event Clustering

The Poisson process assumes that individual claim occurrences are mutually independent. In insurance, this independence assumption is routinely violated:

• Catastrophic Weather Events: A single hailstorm, convective wind event, or winter freeze can generate thousands of comprehensive auto or homeowners claims simultaneously across an entire territory.
• Multi-Claimant Accidents: A single severe multi-vehicle collision can trigger simultaneous bodily injury, property damage, and collision claims from multiple parties.
• Commercial Fleets: Multiple vehicles operating in the same logistics fleet may be exposed to shared hazardous working conditions or poor corporate maintenance policies.

3. Exposure Measurement Inaccuracies

Earned vehicle-years or house-years treat exposure as uniform over time. In reality, miles driven fluctuate drastically by season, remote-work status, or economic conditions, introducing extra-Poisson variance.

---

Diagnosing and Measuring Overdispersion

To detect and quantify overdispersion in a fitted Poisson GLM, actuaries evaluate the Pearson Chi-Square statistic $\chi^2$:

$\chi^2 = \sum_{i=1}^n \frac{(y_i - \hat{\mu}_i)^2}{V(\hat{\mu}_i)} = \sum_{i=1}^n \frac{(y_i - \hat{\mu}_i)^2}{\hat{\mu}_i}$

Where:

• $y_i$ is the observed claim count for policy $i$.
• $\hat{\mu}_i = e_i \exp(\mathbf{x}_i^T \hat{\boldsymbol{\beta}})$ is the fitted expected claim count.
• $V(\hat{\mu}_i) = \hat{\mu}_i$ is the Poisson variance function.

The Pearson dispersion statistic $\hat{\phi}$ is calculated by dividing $\chi^2$ by the residual degrees of freedom:

$\hat{\phi} = \frac{\chi^2}{n - p}$

Where $n$ is the number of observations and $p$ is the number of estimated parameters (including the intercept).

Interpreting the Dispersion Statistic

• $\hat{\phi} \approx 1.0$: Equidispersion holds; the standard Poisson model is adequate.
• $\hat{\phi} > 1.0$ (e.g., $1.5, 2.0, 5.0+$): Significant overdispersion exists.
• $\hat{\phi} < 1.0$: Underdispersion (rare in insurance applications).

Statistical Consequences of Ignoring Overdispersion

When a model exhibits overdispersion, fitting a standard Poisson GLM produces two major statistical consequences:

1. Point Estimates $\hat{\boldsymbol{\beta}}$ Remain Consistent: As long as the conditional mean function $\mathbb{E}[Y \mid \mathbf{x}] = \exp(\mathbf{x}^T \boldsymbol{\beta})$ is correctly specified, maximum likelihood estimates of the regression coefficients remain unbiased and asymptotically consistent.
2. Standard Errors $\text{SE}(\hat{\boldsymbol{\beta}})$ are Underestimated: The standard Poisson model assumes $\phi = 1.0$. If the true dispersion is $\phi = 2.25$, the true standard errors are $\sqrt{2.25} = 1.5$ times larger than reported by the software.

$\text{True } \text{SE}(\hat{\beta}_j) \approx \sqrt{\hat{\phi}} \cdot \text{Reported } \text{SE}(\hat{\beta}_j)$

Underestimating standard errors leads to:

• Artificially inflated $z$-statistics ($z = \hat{\beta} / \text{SE}$).
• Deceptively small $p$-values.
• Artificially narrow confidence intervals.
• Severe Type I error inflation: Variables that have no true relationship with claim frequency will test as "statistically significant" at the $5%$ level, leading to the inclusion of spurious rating variables in rate filings.

---

The Negative Binomial Model (NB2 Parameterization)

The most common method for resolving overdispersion in insurance frequency modeling is the Negative Binomial regression model, specifically the NB2 formulation.

Derivation as a Continuous Poisson-Gamma Mixture

To account for unobserved policyholder heterogeneity, suppose that the conditional claim count for policy $i$, given a latent risk effect $\epsilon_i$, follows a Poisson distribution:

$Y_i \mid \epsilon_i \sim \text{Poisson}(\lambda_i^*)$

Where the hazard rate $\lambda_i^*$ is modulated by an unobserved random multiplier $\epsilon_i$:

$\lambda_i^* = \mu_i \cdot \epsilon_i$

Here, $\mu_i = e_i \exp(\mathbf{x}_i^T \boldsymbol{\beta})$ represents the mean frequency predicted by observable rating variables, and $\epsilon_i$ represents unobserved policyholder risk characteristics.

To ensure that $\mathbb{E}[Y_i] = \mu_i$, we assume that $\epsilon_i$ follows a Gamma distribution with mean 1 and variance $\alpha$:

$\epsilon_i \sim \text{Gamma}\left(\text{shape} = \frac{1}{\alpha}, \text{scale} = \alpha\right)$

$\mathbb{E}[\epsilon_i] = \left(\frac{1}{\alpha}\right) \alpha = 1$ $\text{Var}(\epsilon_i) = \left(\frac{1}{\alpha}\right) \alpha^2 = \alpha$

Where $\alpha > 0$ is the overdispersion parameter.

Deriving the Unconditional Mean and Variance

By the Law of Total Expectation:

$\mathbb{E}[Y_i] = \mathbb{E}_\epsilon [\mathbb{E}_{Y|\epsilon}[Y_i \mid \epsilon_i]] = \mathbb{E}_\epsilon [\mu_i \epsilon_i] = \mu_i \mathbb{E}[\epsilon_i] = \mu_i \cdot 1 = \mu_i$

By the Law of Total Variance:

$\text{Var}(Y_i) = \mathbb{E}_\epsilon [\text{Var}(Y_i \mid \epsilon_i)] + \text{Var}_\epsilon (\mathbb{E}[Y_i \mid \epsilon_i])$

$\text{Var}(Y_i) = \mathbb{E}_\epsilon [\mu_i \epsilon_i] + \text{Var}_\epsilon (\mu_i \epsilon_i)$

$\text{Var}(Y_i) = \mu_i \mathbb{E}[\epsilon_i] + \mu_i^2 \text{Var}(\epsilon_i)$

$\text{Var}(Y_i) = \mu_i + \alpha \mu_i^2$

Key Mathematical Insights of NB2

1. Quadratic Variance Function: In the NB2 model, the variance function is $V(\mu) = \mu + \alpha \mu^2$. The extra-Poisson variance grows quadratically with the mean, providing heavy right-tail flexibility for higher-risk policies.
2. Convergence to Poisson: As the overdispersion parameter $\alpha \to 0$, the quadratic term vanishes:

$\lim_{\alpha \to 0} \text{Var}(Y_i) = \mu_i$

The Negative Binomial distribution converges smoothly back to the Poisson distribution. 3. Likelihood Ratio Test for Overdispersion: To test whether a Negative Binomial model is statistically superior to a Poisson model, actuaries test $H_0: \alpha = 0$ versus $H_1: \alpha > 0$. Because $\alpha = 0$ lies on the boundary of the parameter space, the asymptotic distribution of the Likelihood Ratio Test statistic is a $50:50$ mixture of a point mass at 0 and a $\chi^2_1$ distribution (denoted $0.5 \chi^2_0 + 0.5 \chi^2_1$). The critical value at the $5%$ significance level is $2.706$ (rather than the standard $3.841$).

---

Zero-Inflated Models: ZIP and ZINB

In personal auto and homeowners lines, $90%$ to $98%$ of policies experience zero claims during an annual term. When empirical data exhibit a higher proportion of zeros than predicted by a standard Poisson distribution, analysts often consider Zero-Inflated models:

• Zero-Inflated Poisson (ZIP)
• Zero-Inflated Negative Binomial (ZINB)

The Two-State Mixture Concept

Zero-inflated models posit that the observed zeros arise from two fundamentally different latent mechanisms:

1. Structural Zeros ("Always Zero" State): A subpopulation of policyholders with zero probability of experiencing a loss ($P(Y = 0) = 1.0$). Examples might include a vehicle stored in a secure garage and never driven, or a vacant property with utilities disconnected.
2. Sampling Zeros ("At-Risk" State): Policyholders who are actively exposed to loss, but by chance experienced no claims during the observation period. Their counts follow a standard Poisson or Negative Binomial process.

Mathematical Formulation

Let $\pi_i$ denote the probability of belonging to the structural zero state. The probability mass function of a ZIP model is:

$P(Y_i = 0) = \pi_i + (1 - \pi_i) e^{-\mu_i}$ $P(Y_i = y) = (1 - \pi_i) \frac{\mu_i^y e^{-\mu_i}}{y!}, \quad y \in \{1, 2, 3, \dots\}$

ZIP models require estimating two separate regression equations simultaneously:

1. A logistic regression for the zero-inflation probability: $\text{logit}(\pi_i) = \mathbf{z}_i^T \boldsymbol{\gamma}$.
2. A log-link Poisson regression for the count mean: $\ln(\mu_i) = \mathbf{x}_i^T \boldsymbol{\beta}$.

Why Regulators and Actuaries Prefer Negative Binomial Over ZIP

Despite their statistical elegance, Zero-Inflated models are rarely used in filed P&C rating algorithms for compelling practical and regulatory reasons:

1. Incompatibility with Multiplicative Rating Engines: Standard insurance rating manuals require a single multiplicative chain: $\text{Rate} = \text{Base} \times F_1 \times F_2$. A ZIP model produces two separate sets of non-linear coefficients (one for $\pi_i$ and one for $\mu_i$) that cannot be consolidated into a traditional rating table.
2. The Exposure Scaling Fallacy: In insurance, a short policy period (e.g., a 1-month policy) naturally produces a very high probability of zero claims ($e^{-\mu \cdot (1/12)} \approx 1.0$). A ZIP model misinterprets this short exposure window as evidence that the driver belongs to the "structurally immune" class, distorting rate equity.
3. Regulatory Skepticism: Insurance regulators require clear actuarial justification for rating differentials. Explaining to an insurance department why a policyholder's rate is split into an "immunity likelihood" and an "at-risk severity" invites regulatory objections under unfair discrimination statutes.
4. Occam's Razor: An appropriately specified Negative Binomial GLM with a $\ln(\text{exposure})$ offset naturally handles large proportions of zeros caused by exposure variation and heterogeneity without inventing a fictitious "immune" driver category.

---

Comparison of Frequency Distribution Models