Manage AI/ML Transparency in Data and Algorithm Selection

Transparency Means a Decision Someone Can Reconstruct

ECO Domain I, Task 2 asks you to manage transparency in data selection and algorithm selection. The test is simple: eighteen months from now, when a regulator, an auditor, an angry customer or your own successor asks why this model uses this data and this algorithm, can someone answer from the record rather than from memory? Transparency is a documentation and communication discipline you schedule, not a property the model happens to have.

Explainability Is a Requirement You Set, Not a Property You Discover

Start with the vocabulary, briefly. Glass-box models — logistic regression, shallow decision trees, scorecards — expose their logic directly: you can read the coefficients or follow the rules. Black-box models — deep neural networks, gradient-boosted tree ensembles, large language models — cannot be read the same way and need post-hoc explanation tools added afterwards. Which family a given project ends up using is a Phase 4 technique-selection call, weighed under Domain IV, not settled here.

Your Domain I job sits upstream of that call. You establish, with the stakeholders who will be held to the answer, what has to be explainable before anyone shortlists an algorithm. Run it as a short, specific conversation in Phase 1 Business Understanding, and drive it to three answers for each obligation:

• Who needs the explanation? A declined applicant, a frontline reviewer, a model validator, a regulator and the sponsor each need something different. "The business" is not an audience.
• On what occasion? At every decision, only on an adverse outcome, on appeal, or on audit request. The occasion sets the volume, and volume sets the engineering cost.
• In what form? Reason codes in a letter, top contributing factors on a review screen, a written rationale in the case file, a documented feature-importance report.

Write those answers into the scope statement as a testable acceptance criterion — for example, "every declined application returns the three largest contributing factors in business language" — with a named owner and a Phase 5 test attached. A single sentence of that kind does four jobs at once: it constrains the later technique selection, it sizes the engineering work, it gives the evaluation gate something to verify, and it gives a regulator a written requirement that predates the model. Retrofitting explanations onto a live black box is expensive, occasionally impossible, and always looks reactive.

Document Why You Chose It — and What You Rejected

Documenting model selection criteria and decision rationale means keeping a model selection decision log from Phase 1 through Phase 4. Minimum contents:

• the candidate approaches considered, including the non-AI baseline (business rules, heuristic, existing manual process)
• the criteria and their relative weight: accuracy target, latency, run cost, data required, explainability, maintainability, vendor lock-in
• the option chosen and the measured evidence behind it
• the options rejected and the specific reason each was rejected
• who approved the decision, and when

Rejected options are what convert a preference into a reasoned choice, and an external reviewer will usually ask for them before anything else. A log listing only the winner cannot demonstrate that the choice was reasoned.

Data Provenance and Preprocessing Disclosure

Algorithm transparency is worthless if the data is opaque. For every source feeding the model, require the system of record, the data owner, the extract date and date range, the consent or contract basis, row and field counts, and known quality issues. Then require the preprocessing story: rows excluded and why, missing values imputed and how, outliers treated, features engineered, categories collapsed, sampling or class re-balancing applied, and — critically — how the labels were produced and by whom. Hand-labeled data carries the labelers' assumptions into the model permanently.

A useful rule: if a competent colleague could not rebuild the training set from the documentation, you do not have provenance, you have folklore.

The Two Artifacts to Name

A model card is a short standard disclosure that travels with the model: what it is and its version, intended use, explicitly out-of-scope use, training and evaluation data summaries, performance metrics including subgroup breakdowns, known limitations and failure modes, retraining triggers, and the accountable owner. Produce it in Phase 5 Model Evaluation as an input to the go/no-go decision, then keep it current through Phase 6.

A datasheet for datasets does the same job one level down: why the dataset was created, what it contains, how it was collected, how it was preprocessed, permitted uses, and who maintains it. Requiring a datasheet from a vendor before buying third-party data is one of the cheapest controls available to you.

Global vs Local Explanations

Two different questions, two different tools.

Global explanations describe overall behavior: which features matter most across the whole population, and in which direction. Feature importance rankings and partial dependence views answer "what drives this model?" — the question a sponsor, model validator or risk committee asks.

Local explanations describe one prediction: why this applicant, this claim, this transaction. SHAP (SHapley Additive exPlanations) attributes a specific prediction across its input features relative to a baseline average, answering which inputs pushed this case up or down and by roughly how much; it is consistent but computationally heavier. LIME (Local Interpretable Model-agnostic Explanations) fits a simple approximate model around the single case — fast and model-agnostic, but an approximation that can shift between runs.

You need none of the mathematics. You need three things: SHAP and LIME answer the individual case question, feature importance answers the overall behavior question, and neither proves the model is correct or fair. They describe; they do not validate.

Match the Explanation to the Audience

Test the Criterion Before the Gate, Then Disclose the Result

A criterion nobody verifies is a wish. Ahead of the Phase 5 go/no-go, pull a sample of real cases spanning the outcome range and check three things. Is an explanation produced at all for every case the criterion covers, including the awkward ones — missing inputs, borderline scores, manual overrides? Is it intelligible to the audience it was written for, confirmed by a person from that audience rather than by the data science team? And is it stable, so that two runs on the same case do not return contradictory reasons? Record the answers in the model card and in the go/no-go memo, together with the cases where the explanation failed and what was done about them. That record is what turns "the model is explainable" from an assertion into a finding someone else can check.

Audit Trails and Human Review

Maintaining audit trails for algorithmic decision-making means logging, per decision: model version, input values or a hash of them, output score, threshold applied, action taken, the explanation shown, and any human override with its stated reason. Retain it for at least the complaint and regulatory window that applies to the business process.

Human-in-the-loop review is itself a transparency control, but only if reviewers receive a usable case-level explanation. A reviewer handed a bare score rubber-stamps, producing the appearance of oversight and none of the substance. Track override rates: a rate near zero usually means rubber-stamping, and a very high rate means the model is not trusted. Both are findings you take to the sponsor rather than numbers you file.