2.4 Data Quality Under ASOP 23 & Modeling Under ASOP 56

Why Two ASOPs Appear in a Predictive Analytics Syllabus

The PCPA Content Outline lists only two Actuarial Standards of Practice among its readings, and both are attached to Domain A, Dealing with Data:

• ASOP No. 23, Data Quality (December 2016)
• ASOP No. 56, Modeling (December 2019)

They are there because the data domain is not only a technical skill. The same decisions — accept this extract, recode that field, exclude these records — are professional judgements that an actuary must be able to defend.

ASOP 23: Data Quality

ASOP 23 applies when an actuary selects data, relies on data supplied by others, prepares data, or reviews data used in an actuarial work product.

Selecting the data

The actuary should consider whether the data is appropriate for the intended purpose, and weigh the cost and feasibility of obtaining alternative data against the benefit. Perfect data is rarely available; the standard asks for a reasoned choice, not an impossible one.

Reviewing the data

The actuary should review the data for appropriateness, reasonableness and comprehensiveness. The standard is explicit that this review is not an audit. The actuary is not obliged to independently verify every record, but is obliged to look — and a reviewer who never looked cannot claim the exemption.

In a P&C predictive modelling context, that review is the EDA routine: reconciliation to source totals, missingness by field and segment, range and sentinel checks, duplicate-key checks, and consistency of definitions across policy years.

Handling defects

If a review reveals a material defect, the actuary should consider whether it can be corrected or its effect adjusted for. Where it cannot, ASOP 23 requires the actuary either to disclose the limitation and its likely effect, or, if the defect is severe enough, to state that the data is not adequate for the intended purpose.

Disclosure

Actuarial reports that rely on data should disclose the source of the data, any reliance on data supplied by others, any material judgements or adjustments the actuary made, and any unresolved concerns about the data. On the PCPA Project, that is a sentence or two in the technical report — and it is also a rubric criterion.

ASOP 56: Modeling

ASOP 56 applies when an actuary designs, develops, selects, modifies, uses, reviews, or relies on a model. It is the standard that frames the rest of PCPA.

Its core expectations:

ASOP 56 is also the standard that makes "I ran the package and it converged" an inadequate answer. If you cannot explain what the model assumes and where it breaks, you have not met the standard.

The Data Defects These Standards Exist to Catch

Target leakage

A predictor that is only populated after the outcome occurred. In P&C, the common offenders are claim counts, paid or incurred loss, litigation and salvage flags, adjuster-assigned severity codes, and audited exposure. Each correlates powerfully with the target and none of them is available when the model must score a new risk.

The symptom is unmistakable: implausibly good validation performance, often with a single dominant coefficient. If a model's holdout Gini jumps from 0.30 to 0.85 when one variable is added, suspect leakage before celebrating.

Look-ahead bias

A subtler cousin. The field itself would exist at decision time, but the value you used was computed with information from the whole data set — a territory relativity computed on all years including the holdout, a target-encoded agency mean computed before the train/test split, an inflation adjustment applied with hindsight. The fix is to derive any data-dependent quantity inside the training fold only.

Duplicate and near-duplicate records

System conversions, mid-term endorsements and re-issues produce multiple rows for what is economically one exposure. Left uncorrected they multiply exposure and depress frequency. Endorsement handling in particular must be decided explicitly: does an endorsement replace the original row, or add an incremental one?

Inconsistent definitions across time

The most expensive defect to find late. A field's meaning changes when a system is replaced, a coverage is re-coded, or a claim department changes its case-reserving philosophy. The signature is a step change in a one-way plot at a specific date rather than a smooth trend. Plot every important field by policy month before trusting it across years.

Inconsistent bases

Mixing paid and incurred losses, accident-year and policy-year records, gross and net of reinsurance, or earned and written exposure. Each mixture produces a model that is internally incoherent and externally indefensible.

[!WARNING] A useful screen before fitting anything: for each candidate predictor, write the date on which its value becomes known. If that date is later than the date the model must produce a prediction, the field is out — regardless of how much lift it shows.

Carrying This Into the Report

The project rubric asks the candidate to describe any anomalous characteristics of the data and how they were addressed. That is ASOP 23 disclosure expressed as a grading criterion. Two or three sentences naming the defect, the treatment, and the residual limitation satisfies it — and a report with no such sentences reads as though no review was performed.