3.4 Modeling Techniques for Interpretability Requirements
Key Takeaways
Intrinsically interpretable models — linear, logistic, shallow trees, GAMs, rule lists — are the answer when a regulator or reviewer must understand the model itself, not just its outputs.
Post-hoc explanations such as Shapley values, integrated gradients, and LIME describe a black box approximately and do not make it inherently interpretable.
Integrated gradients requires a differentiable model and a baseline input; sampled Shapley works on non-differentiable models such as tree ensembles.
Global interpretability answers "what drives this model", local interpretability answers "why this decision" — regulatory adverse-action notices need the local kind.
Monotonic constraints let a boosted tree encode required directional relationships, giving auditable behaviour without dropping to a linear model.
3.4 Modeling Techniques for Interpretability Requirements
Blueprint reference: Section 3.1, "Modeling techniques given interpretability requirements."
Interpretability appears in Section 3 as a modeling consideration, not only as an explainability tool bolted on afterwards. The exam distinction that matters most: some requirements can be met with post-hoc explanations, and some can only be met by choosing a different model.
Intrinsic Interpretability Versus Post-Hoc Explanation
| Intrinsically interpretable | Post-hoc explained | |
|---|---|---|
| What it means | The model's structure is directly readable | A separate method approximates the model's behaviour |
| Examples | Linear/logistic regression, shallow decision trees, GAMs, rule lists, scorecards | Shapley values, integrated gradients, LIME, partial dependence |
| Fidelity | Exact — the explanation is the model | Approximate — a faithful-but-not-identical account |
| Regulator asks "how does the model decide?" | Answerable exactly | Answerable only approximately |
| Typical accuracy | Lower on complex problems | Whatever the underlying model achieves |
The decisive scenario: a regulator or model-risk committee requires that the model itself be reviewable and its decision logic reproducible by hand. Post-hoc attributions do not satisfy that, because they are an approximation produced by a second algorithm. Choose a linear model, a scorecard, or a shallow tree.
When the requirement is instead "explain individual decisions to affected customers," a black-box model with reliable local attributions is usually acceptable, and the accuracy is worth keeping.
Global Versus Local
- Global — which features drive the model overall. Feature importances, aggregate attributions, partial dependence plots. Used for model review, debugging, and documentation.
- Local — why this input received this output. Per-row attributions. Used for adverse-action notices, dispute handling, and clinician-facing justification.
Regulatory scenarios almost always need local explanations. "We publish a ranked list of the model's top features" is not an adverse-action notice, and it is a common wrong option.
The One Applicability Rule to Carry Into Design
The mechanics of Explainable AI on the Agent Platform — sampled Shapley, integrated gradients, XRAI, and baseline selection — are covered in Section 6.2, Responsible AI, Explainable AI, Fairness Metrics and Model Governance. Only one of those facts constrains the modeling decision and therefore belongs here:
A non-differentiable model cannot be explained with gradient-based methods. Integrated gradients and XRAI need gradients, which a tree ensemble does not have, so choosing gradient boosting commits you to model-agnostic sampled Shapley for attributions. Choosing a neural network keeps both options open. If a project has already standardized on integrated gradients for consistency across its model portfolio, that decision silently rules out tree ensembles — which is exactly the kind of constraint that should be surfaced at design time rather than discovered at explanation time.
Monotonic Constraints: The Middle Path
Often the requirement is not full interpretability but guaranteed directional behaviour: risk must not decrease as debt rises; price must not decrease as square footage rises. Gradient boosting implementations support monotonic constraints that enforce this per feature during training.
This gives an auditable, defensible property — "the model provably never rewards higher debt" — while keeping most of the accuracy of a boosted ensemble. When a scenario asks for a guarantee about direction rather than a full account of the logic, monotonic constraints beat both a linear model (accuracy loss) and unconstrained boosting (no guarantee).
Other Techniques That Buy Interpretability
- Feature reduction. A model with 12 well-chosen features is reviewable; one with 900 is not, whatever the attribution method.
- Binned / scorecard models. Widely used in credit because the resulting model is a small table a human can read and audit.
- Generalized additive models (GAMs). Each feature contributes an independently plottable shape function, giving non-linearity with per-feature readability.
- Distillation to a simpler surrogate. Train a complex model, then fit an interpretable one to its predictions. Useful for understanding, but the surrogate is not the deployed model — do not offer it as an audit substitute for the deployed system.
Documenting the Decision
Whatever technique is chosen, the artifacts a review expects are the model card (intended use, data, evaluation, limitations), the explanation method and its configuration (including baseline), and evidence that explanations were validated — for example, that attributions align with domain expectations on known cases.
Exam Traps
- Offering post-hoc attributions when the requirement is that the model itself be reviewable.
- Integrated gradients on a boosted tree. Not differentiable; use sampled Shapley.
- Forgetting the baseline for integrated gradients, or choosing a meaningless one.
- Global importances presented as per-customer explanations.
- A distilled surrogate offered as the auditable artifact when a different model is deployed.
A banking regulator requires that the credit decisioning model itself be reviewable, with decision logic that a human reviewer can reproduce by hand from the model artifact. The team proposes a gradient boosted ensemble with sampled Shapley explanations. Why is this insufficient?
Sampled Shapley is too slow to compute for every application within the decisioning latency budget.
Shapley values are an approximate post-hoc account of a black box, so the model's own logic remains unreviewable; the requirement calls for an intrinsically interpretable model such as a scorecard, linear model, or shallow tree.
Gradient boosted ensembles cannot produce local explanations, only global feature importances.
Shapley values require a differentiable model, which a boosted tree is not.
A team needs per-row attributions for a deployed XGBoost model that scores loan applications. An engineer configures integrated gradients and the configuration is rejected. What is the correct method and why?
Partial dependence plots, which provide the per-applicant attributions the requirement needs.
Integrated gradients with an all-zeros baseline, which is valid once a baseline is supplied.
Sampled Shapley, because integrated gradients requires a differentiable model and a tree ensemble has no gradients.
XRAI, which generalizes integrated gradients to non-differentiable models.
A pricing model must provably never predict a lower price as square footage increases, but the business also wants the accuracy of a boosted ensemble. Which technique satisfies both?
Train the boosted ensemble normally and reject predictions at serving time that violate the expected direction.
Replace the ensemble with a linear regression so the coefficient sign guarantees the direction.
Train the ensemble and validate the direction with partial dependence plots on the evaluation set.
Train the boosted ensemble with a monotonic constraint on the square footage feature.
A hospital deploys a deep neural network for risk scoring and must give each clinician a per-patient explanation. The team configures integrated gradients but attributions look erratic and clinically implausible. What is the most likely configuration problem?
The chosen baseline input is inappropriate, since integrated gradients produces attributions relative to a baseline and a meaningless baseline yields misleading results.
Integrated gradients only supports image inputs, so a tabular clinical model requires XRAI instead.
The attributions should be aggregated across the training set, because integrated gradients is a global method.
The model needs to be retrained with monotonic constraints so attributions become directionally stable.
Sections you finish are checked off in the contents.