9.2 Explaining Why a Variable Was or Was Not Included
Key Takeaways
- The project rubric contains a criterion devoted solely to describing why a variable was or was not included in the model.
- Inclusion decisions rest on five grounds: statistical evidence, materiality, stability, availability at decision time, and business or regulatory acceptability.
- Statistical significance alone is neither necessary nor sufficient; a significant but immaterial variable and a credible but marginally significant one both need a stated rationale.
- Variables dropped for correlation, thin data, leakage or instability should be named in the report, because an unmentioned variable reads as an oversight rather than a decision.
- A short inclusion table listing each candidate predictor, the decision, and the one-line reason is the most word-efficient way to satisfy the criterion.
A Criterion of Its Own
Among the twelve published performance criteria for the PCPA Project, one reads simply: "Candidate describes why a variable was or was not included in the model." It sits under Model Interpretation and Presentation (C-2), the domain carrying 40% of the project rubric.
This is not a request for a p-value table. It is a request for reasoning — including reasoning about the variables that did not make the final model. A predictor that vanishes without comment looks like something the candidate forgot, not something they decided.
Five Grounds for the Decision
Every inclusion or exclusion rests on at least one of these. Naming which one applies is what turns an assertion into a justification.
1. Statistical evidence. Does the variable improve the model? The usual evidence is a drop in deviance relative to the degrees of freedom spent, an improvement in AIC or BIC, a likelihood ratio test on nested models, and coefficients that are large relative to their standard errors.
2. Materiality. Does the effect matter in dollars? A variable whose relativities span 0.99 to 1.01 is not worth a rating dimension even if its p-value is vanishingly small. State the relativity range, not only the significance.
3. Stability. Does the effect hold up? A coefficient that reverses sign between cross-validation folds, or between policy years, or when a correlated predictor is removed, is not a finding. Instability is a legitimate and well-regarded reason to exclude.
4. Availability at decision time. Will the field exist when the model must score a risk? A predictor populated only after a claim, or missing for 70% of new business, cannot carry a relativity regardless of its statistical strength.
5. Business and regulatory acceptability. Is the variable usable, explainable, and permitted in the jurisdiction? A predictor whose direction contradicts well-established risk understanding invites a question that the model must be able to answer.
Phrasing Each Decision
One sentence per variable is usually enough. The structure is decision + ground + evidence.
| Situation | Wording |
|---|---|
| Included, strong and material | "Vehicle age was retained: banded into five levels it reduced deviance by 310 on 4 d.f. and produces relativities from 0.82 to 1.35." |
| Included despite marginal significance | "Prior-term claim indicator was retained despite a p-value of 0.06: the effect is consistent in direction across all folds and is a well-established frequency predictor." |
| Excluded, immaterial | "Payment plan was dropped: although significant at this sample size, its relativities span 0.99-1.01 and it adds a rating dimension for no material price difference." |
| Excluded, correlated | "Vehicle value was dropped in favour of vehicle symbol: the two correlate at 0.97 and including both produced offsetting coefficients with inflated standard errors." |
| Excluded, unstable | "Agency tenure was dropped: its coefficient changed sign between cross-validation folds, indicating the effect is not stable." |
| Excluded, thin data | "Five construction classes with fewer than 200 exposure units were grouped into 'other' rather than carrying their own coefficients, because a GLM assigns full credibility to every level." |
| Excluded, unavailable | "Audited payroll was excluded: it is only known after the policy period and cannot be observed when a new-business quote is rated." |
The Inclusion Table
With 1,250 words to spend, prose covering fifteen candidate predictors is unaffordable. A table is dramatically more efficient, and it also reads as systematic rather than selective.
| Candidate predictor | Decision | Reason |
|---|---|---|
| Driver age | Included, 5 bands | U-shaped one-way; deviance -214 on 4 d.f. |
| Territory | Included, 4 groups | Material; thin levels grouped |
| Vehicle symbol | Included | Material; retained over correlated vehicle value |
| Vehicle value | Excluded | Correlated 0.97 with symbol; unstable when both included |
| Prior claims | Included | Consistent across folds; established predictor |
| Payment plan | Excluded | Immaterial: relativity range 0.99-1.01 |
| Audited payroll | Excluded | Not available at quote |
| Litigation flag | Excluded | Post-outcome field; target leakage |
Eight decisions in roughly sixty words, and every one carries a ground. This single exhibit can satisfy the criterion on its own, leaving the narrative free for the two or three decisions that need more explanation.
Why Stepwise Output Is Not a Justification
"The variables in the final model were selected by backward stepwise elimination using AIC" describes a procedure, not a rationale. It tells a grader nothing about materiality, stability, availability or acceptability, and automated selection procedures are well known to be unstable when predictors are correlated.
If an automated procedure was used, say so — and then say what you did with its output. "Backward stepwise on AIC produced an initial candidate set; two of its retained variables were then removed after VIF review, and one variable it dropped was reinstated because it is a standard rating dimension with a stable coefficient" is a rationale. The bare procedure is not.
[!WARNING] Never justify inclusion solely by predictive lift. A variable can improve a holdout Gini and still be unusable because it will not exist at scoring time, because it is unstable, or because it is not permitted in the jurisdiction. Lift is evidence on ground 1 only; the criterion asks the candidate to have considered the rest.
A candidate's report lists the final model's variables and their coefficients but never mentions the six candidate predictors that were tested and dropped. Which rubric criterion is unmet?
Which justification for excluding a predictor is weakest as written?
A predictor is significant at p < 0.001 in a 400,000-record data set, its relativities range from 0.99 to 1.01, and it would add a new rating dimension. What is the best-supported decision and rationale?