3.4 Modeling Techniques for Interpretability Requirements
Key Takeaways
- Intrinsically interpretable models — linear, logistic, shallow trees, GAMs, rule lists — are the answer when a regulator or reviewer must understand the model itself, not just its outputs.
- Post-hoc explanations such as Shapley values, integrated gradients, and LIME describe a black box approximately and do not make it inherently interpretable.
- Integrated gradients requires a differentiable model and a baseline input; sampled Shapley works on non-differentiable models such as tree ensembles.
- Global interpretability answers "what drives this model", local interpretability answers "why this decision" — regulatory adverse-action notices need the local kind.
- Monotonic constraints let a boosted tree encode required directional relationships, giving auditable behaviour without dropping to a linear model.
3.4 Modeling Techniques for Interpretability Requirements
Blueprint reference: Section 3.1, "Modeling techniques given interpretability requirements."
Interpretability appears in Section 3 as a modeling consideration, not only as an explainability tool bolted on afterwards. The exam distinction that matters most: some requirements can be met with post-hoc explanations, and some can only be met by choosing a different model.
Intrinsic Interpretability Versus Post-Hoc Explanation
| Intrinsically interpretable | Post-hoc explained | |
|---|---|---|
| What it means | The model's structure is directly readable | A separate method approximates the model's behaviour |
| Examples | Linear/logistic regression, shallow decision trees, GAMs, rule lists, scorecards | Shapley values, integrated gradients, LIME, partial dependence |
| Fidelity | Exact — the explanation is the model | Approximate — a faithful-but-not-identical account |
| Regulator asks "how does the model decide?" | Answerable exactly | Answerable only approximately |
| Typical accuracy | Lower on complex problems | Whatever the underlying model achieves |
The decisive scenario: a regulator or model-risk committee requires that the model itself be reviewable and its decision logic reproducible by hand. Post-hoc attributions do not satisfy that, because they are an approximation produced by a second algorithm. Choose a linear model, a scorecard, or a shallow tree.
When the requirement is instead "explain individual decisions to affected customers," a black-box model with reliable local attributions is usually acceptable, and the accuracy is worth keeping.
Global Versus Local
- Global — which features drive the model overall. Feature importances, aggregate attributions, partial dependence plots. Used for model review, debugging, and documentation.
- Local — why this input received this output. Per-row attributions. Used for adverse-action notices, dispute handling, and clinician-facing justification.
Regulatory scenarios almost always need local explanations. "We publish a ranked list of the model's top features" is not an adverse-action notice, and it is a common wrong option.
The One Applicability Rule to Carry Into Design
The mechanics of Explainable AI on the Agent Platform — sampled Shapley, integrated gradients, XRAI, and baseline selection — are covered in Section 6.2, Responsible AI, Explainable AI, Fairness Metrics and Model Governance. Only one of those facts constrains the modeling decision and therefore belongs here:
A non-differentiable model cannot be explained with gradient-based methods. Integrated gradients and XRAI need gradients, which a tree ensemble does not have, so choosing gradient boosting commits you to model-agnostic sampled Shapley for attributions. Choosing a neural network keeps both options open. If a project has already standardized on integrated gradients for consistency across its model portfolio, that decision silently rules out tree ensembles — which is exactly the kind of constraint that should be surfaced at design time rather than discovered at explanation time.
Monotonic Constraints: The Middle Path
Often the requirement is not full interpretability but guaranteed directional behaviour: risk must not decrease as debt rises; price must not decrease as square footage rises. Gradient boosting implementations support monotonic constraints that enforce this per feature during training.
This gives an auditable, defensible property — "the model provably never rewards higher debt" — while keeping most of the accuracy of a boosted ensemble. When a scenario asks for a guarantee about direction rather than a full account of the logic, monotonic constraints beat both a linear model (accuracy loss) and unconstrained boosting (no guarantee).
Other Techniques That Buy Interpretability
- Feature reduction. A model with 12 well-chosen features is reviewable; one with 900 is not, whatever the attribution method.
- Binned / scorecard models. Widely used in credit because the resulting model is a small table a human can read and audit.
- Generalized additive models (GAMs). Each feature contributes an independently plottable shape function, giving non-linearity with per-feature readability.
- Distillation to a simpler surrogate. Train a complex model, then fit an interpretable one to its predictions. Useful for understanding, but the surrogate is not the deployed model — do not offer it as an audit substitute for the deployed system.
Documenting the Decision
Whatever technique is chosen, the artifacts a review expects are the model card (intended use, data, evaluation, limitations), the explanation method and its configuration (including baseline), and evidence that explanations were validated — for example, that attributions align with domain expectations on known cases.
Exam Traps
- Offering post-hoc attributions when the requirement is that the model itself be reviewable.
- Integrated gradients on a boosted tree. Not differentiable; use sampled Shapley.
- Forgetting the baseline for integrated gradients, or choosing a meaningless one.
- Global importances presented as per-customer explanations.
- A distilled surrogate offered as the auditable artifact when a different model is deployed.
A banking regulator requires that the credit decisioning model itself be reviewable, with decision logic that a human reviewer can reproduce by hand from the model artifact. The team proposes a gradient boosted ensemble with sampled Shapley explanations. Why is this insufficient?
A team needs per-row attributions for a deployed XGBoost model that scores loan applications. An engineer configures integrated gradients and the configuration is rejected. What is the correct method and why?
A pricing model must provably never predict a lower price as square footage increases, but the business also wants the accuracy of a boosted ensemble. Which technique satisfies both?
A hospital deploys a deep neural network for risk scoring and must give each clinician a per-patient explanation. The team configures integrated gradients but attributions look erratic and clinically implausible. What is the most likely configuration problem?