5.3 Explainability, Interpretability & Transparency Methods (LIME, SHAP, Datasheets)
Key Takeaways
- Interpretability refers to intrinsically understandable model architectures (e.g., decision trees), whereas explainability uses post-hoc methods for complex models (e.g., deep neural networks).
- SHAP (SHapley Additive exPlanations) provides game-theoretic feature attribution grounded in Shapley values, ensuring local accuracy and consistency.
- LIME (Local Interpretable Model-agnostic Explanations) builds surrogate linear models around individual predictions to explain complex model behavior locally.
- Datasheets for Datasets standardizes documentation regarding dataset motivation, composition, collection process, preprocessing, and recommended uses.
- Model Cards document model architecture, intended usage, evaluation metrics, fairness audits, and operational boundaries for transparency.
5.3 Explainability, Interpretability & Transparency Methods (LIME, SHAP, Datasheets)
ISO/IEC 42001 emphasizes that AI systems must operate transparently, allowing interested parties to understand, audit, and evaluate how automated decisions are generated. Explaining complex AI systems—particularly deep neural networks and ensemble architectures—is a core requirement for risk management, regulatory compliance (such as the GDPR right to explanation), and building stakeholder trust.
Distinguishing Interpretability, Explainability, and Transparency
In AI governance terminology, these three terms refer to distinct dimensions of system legibility:
- Transparency: The macro-level availability of information regarding an AI system's design, training data, operational boundaries, governance controls, and performance metrics.
- Interpretability (Intrinsic Transparency): The degree to which a human can comprehend the internal decision logic of an AI model directly from its architecture without external mathematical surrogate tools.
- Explainability (Post-Hoc Transparency): The capability to generate human-understandable explanations for specific predictions made by complex, non-interpretable ("black-box") models using external diagnostic frameworks.
Intrinsic vs. Post-Hoc Explainability
| Model Category | Transparency Type | Technical Examples | Governance & Audit Implications |
|---|---|---|---|
| Intrinsic / Glass-box | High natural interpretability | Linear Regression, Decision Trees (shallow), Generalized Additive Models (GAMs) | Easy to audit directly; decision logic is self-evident; lower predictive capacity on complex non-linear data |
| Post-hoc Local Explanations | Explains single individual predictions | LIME, SHAP, TreeSHAP, Integrated Gradients | Suitable for explaining specific automated decisions to end-users and regulators |
| Post-hoc Global Explanations | Explains overall model feature importance | Global SHAP summary plots, Partial Dependence Plots (PDP), Permutation Feature Importance | Essential for model validation, auditing global feature dependence, and verifying system safety |
Feature Attribution Frameworks: SHAP and LIME
Post-hoc local explainability relies heavily on feature attribution methods, which assign an importance score to each input feature for a given prediction.
| Feature Attribution Method | Theoretical Basis | Scope | Strengths | Operational Drawbacks |
|---|---|---|---|---|
| SHAP (SHapley Additive exPlanations) | Game theory (Shapley values from cooperative games) | Local and Global | Solid mathematical foundation; guarantees efficiency, symmetry, dummy, and additivity properties | Computationally expensive for high-dimensional models; requires background sample approximation |
| LIME (Local Interpretable Model-agnostic Explanations) | Local linear surrogate modeling | Local | Fast computation; completely model-agnostic; intuitive local linear approximation | Sampling instability; explanations can vary between identical runs due to local perturbation sampling |
| Integrated Gradients | Axiomatic path integrals of gradients | Local | Computationally efficient for deep neural networks; satisfies completeness axiom | Requires access to model gradients (white-box access); not applicable to decision trees |
Deep-Dive: SHAP (Shapley Additive Explanations)
SHAP computes the marginal contribution of each feature across all possible feature subsets (coalitions). The Shapley value $\phi_i$ for feature $i$ is calculated as: This game-theoretic foundation guarantees that the sum of feature attributions equals the difference between the model output and the expected baseline prediction (Efficiency property).
Deep-Dive: LIME
LIME generates synthetic perturbations around a target instance $x$, obtains model predictions for these perturbed samples, and trains an interpretable surrogate model $g \in G$ (such as a sparse linear model) by minimizing a loss function $\mathcal{L}(f, g, \pi_x)$ weighted by proximity measure $\pi_x$, plus a complexity penalty $\Omega(g)$.
Standardized Artifact Documentation: Datasheets for Datasets
Structural transparency requires standardized documentation for datasets and models. Datasheets for Datasets (Gebru et al.) provides a structured questionnaire framework covering key lifecycle stages:
- Motivation: Who created the dataset, why was it created, and who funded it?
- Composition: What do the instances represent? Are there missing data or sensitive features?
- Collection Process: How was data acquired, what sampling mechanisms were used, and was consent obtained?
- Preprocessing / Cleaning: How was raw data transformed, normalized, or anonymized?
- Uses: What tasks is the dataset suitable for, and what uses are explicitly discouraged?
- Distribution & Maintenance: How is the dataset distributed, updated, and maintained?
Model Cards and System Transparency Artifacts
Model Cards (Mitchell et al.) complement dataset documentation by standardizing operational disclosure for trained model artifacts:
- Model Details: Developer, model release date, version, architecture, and type.
- Intended Use: Primary intended applications, out-of-scope use cases, and target user personas.
- Factors: Demographic, environmental, or technical factors evaluated for performance disparities.
- Metrics: Evaluation measures (Accuracy, F1-score, ROC-AUC, Fairness metrics) and selection rationale.
- Evaluation Data: Datasets used for validation and testing, including representation details.
- Ethical Considerations & Caveats: Operational limitations, sensitive edge cases, and known risks.
Governance Integration for ISO/IEC 42001 Compliance
Lead Implementers must operationalize explainability controls within the AIMS:
- Mandate Datasheets for Datasets for all ingested data assets.
- Require Model Cards prior to promoting models from staging to production.
- Embed automated SHAP / LIME explanation generators into decision workflows requiring explainable outputs (e.g., credit underwriting, medical diagnostics).
Which explainability framework is grounded in cooperative game theory and guarantees that the sum of individual feature attributions equals the difference between the model output and the baseline expected prediction?
What is a primary operational drawback of using LIME (Local Interpretable Model-agnostic Explanations) for post-hoc model explainability?
Which standardized documentation artifact, developed by Gebru et al., systematically records dataset motivation, composition, collection methodology, preprocessing, and recommended usage boundaries?