Manage Accountability Documentation and the Audit Trail
Key Takeaways
- A decision record answers four questions — what was decided, by whom, on what evidence, and what alternative was rejected and why; the fourth is the one teams skip and auditors ask for.
- Every accountability record must point at the specific artifacts behind a result — the dataset snapshot, the training run, the model version — by identifier rather than describe them in prose; the technical pinning discipline itself belongs to Domain IV model QA/QC.
- Every CPMAI phase ends in a go/no-go, and each needs a signed record naming the approver and their authority, the evidence reviewed, and every condition attached with an owner and due date.
- Chain of custody for training and test data covers origin, every hand that touched it, and the license or consent that permits model training specifically — permission to analyze is not permission to train.
- The reproducibility test: could an independent auditor rebuild this model result 18 months from now from your records alone, with no access to the project team?
Documentation Is the Deliverable That Proves Everything Else Happened
The other Domain I tasks produce claims: we protected the data, we made the model explainable, we tested for bias. Task 5 produces the evidence. Treat the decision log as a first-class deliverable with an owner, a template, and a slot in the schedule — not something written up at the end. (Keep it distinct from Domain V, Task 3 model governance, which supervises the model after go-live — drift, retraining, decommissioning — while accountability documentation records how it was built and who approved it.)
What a Decision Record Actually Contains
Four fields, every time:
- What was decided — stated as a decision, not a status.
- By whom — a named person with the role and the authority to make it.
- On what evidence — the analysis, SME opinion, or test result relied on, with a pointer to it.
- What was rejected and why — the alternative considered and the reason it lost.
The fourth is the one teams skip and auditors ask for. "We trained on 18 months of history" is a status. "We trained on 18 months because the 2023 pricing change made earlier data non-comparable; the 36-month option was rejected for that reason, confirmed by the pricing SME on 4 March" is a record.
Decisions worth logging: the problem framing, the target label definition, the inclusion or exclusion of a data source, the fairness metric and groups tested, the decision threshold, the accept/reject on a candidate modeling technique, and every scope change.
Point the Record at the Artifacts, Not at a Description
The enabler behind this task calls for version control over models, data, and training processes. The technical discipline that delivers it — pinning the dataset snapshot, the code commit, the configuration, the seed, and the environment — is covered under Domain IV model QA/QC, so learn the pinning rules there. What Domain I asks of you sits one step away from the tooling and is the part teams miss: every accountability record must point at those artifacts by identifier rather than describe them in prose.
In practice that means a decision record cites the run it relied on, a gate approval cites the evaluation report and the exact candidate it cleared, and a deployment record cites the model version together with the dataset snapshot behind it. "Trained on the customer table" is a description; a snapshot identifier with an as-of timestamp is a reference someone else can follow two years later. Your own test is a walk: take the model that is live today and see whether the file lets you step from the deployed version to the run that produced it, from the run to the data it consumed, and from that data to the permission that allowed its use. You do not operate the tools that store those identifiers, but you require them in the plan, budget the effort, and spot-check that the fields you will one day be asked for are actually populated. A spreadsheet is acceptable at small scale only if each row still resolves to a specific run.
Signed Approvals at the Go/No-Go Gates
CPMAI is phased, and each phase ends in a go/no-go decision. A signed approval must exist for each. Record the gate, date, decision, the named approver and their authority, the evidence reviewed, every condition with an owner and due date, and any dissent. The classic failure is the conditional go whose conditions are never written down: six months later the record reads as unconditional approval. "The steering committee approved it", with no name, date, or artifact list, is not an approval record.
Chain of Custody for Training and Test Data
Three questions, answered in writing for every dataset:
- Where did it come from? Source system, data owner, extraction date, extraction method.
- Who touched it? Every transfer, transformation, and access grant between the source and the training run.
- What permits its use? The contract clause, consent text, or terms of service — including purpose limits, expiry, and specifically whether model training is permitted, which is not the same as permission to analyze or report.
Test and holdout data need their own custody trail: the entire value of a holdout is that the model never saw it, provable only from records. The recurring trap is a purchased or scraped dataset whose license covers "internal analysis" being quietly read as covering model training.
The Accountability Artifact Set
| Artifact | What it records | Who owns it | When it is produced |
|---|---|---|---|
| Decision log | What was decided, by whom, on what evidence, what was rejected and why | Project manager | Continuously, every phase |
| Data inventory and source register | Each source, its owner, extraction date, and the license or consent that permits use | Project manager with the data SME | Data Understanding |
| Chain-of-custody record | Every transfer, transformation, and access grant for training and test data | Data engineering lead; PM verifies completeness | Data Understanding into Data Preparation |
| Experiment tracking run record | Parameters, dataset version, code commit, metrics, and artifacts for each run | Data science lead | Model Development and Model Evaluation |
| Model registry entry | Which run became a candidate, which was approved, current stage, what it supersedes | Model owner with MLOps | Model Evaluation into Model Operationalization |
| Model documentation (model card) | Purpose, training data, performance including by group, known limits, prohibited uses | Model owner; PM reviews | Before the operationalization gate |
| Signed go/no-go memo | Gate, decision, named approver and authority, evidence reviewed, conditions, dissent | Project manager; signed by the gate authority | End of each CPMAI phase |
| Accountability report | Readable synthesis for executives and regulators, with pointers into the evidence | Project manager | At operationalization and on a set review cycle |
The Reproducibility Test
Apply one question to the whole file: could an independent auditor, 18 months from now, with no access to anyone who worked on the project, rebuild this result from your records alone? Walk the chain — the deployed model version, the run that produced it, the exact data snapshot, the transformations, the configuration, the evaluation results, and the approval with its conditions. Any break is an audit finding waiting to happen. Run the test yourself at the operationalization gate, while the people who know the answers are still around.
Preparing the Accountability Report
Executives and regulators do not read the decision log; they read a synthesis that points into it. Write for a reader with no project context, covering: what the system does and which decision it influences; what data it was built on and under what authority; how it performed, including by group; known limitations and where it must not be used; who approved it and when; what monitoring and human-review path exists; and where the evidence is stored. If the report cannot be read on its own, it is a log extract, not an accountability report.
Eighteen months after deployment, a regulator asks your organization to explain why the fraud-detection model began rejecting a particular category of transactions. The team can produce the training script from source control and the deployed model file, but cannot reproduce the original result. What is the most likely gap in the accountability documentation?
At the end of Model Evaluation, the steering committee issues a 'conditional go' — production deployment may proceed provided the false-positive rate is re-measured on the next quarter's data and an appeals path is added. What must the project manager record?
An external auditor reviewing a credit-risk model asks the team to demonstrate that the holdout test set was never available to the training pipeline. The team is confident the split was honored, but there is no record of when the split was made, which snapshot it was drawn from, or who has had access to the holdout since. What should the project manager do?