Oversee Data Evaluation
Key Takeaways
- Data evaluation quality must be reported per dimension with a method and a threshold; 'the data is complete' is not a finding, '82% of records carry income, 61% in the broker channel' is.
- Missingness concentrated in one channel, region, or customer group is missing-not-at-random — a bias risk, not merely a gap — so always demand missingness broken out by segment.
- Freshness asks whether data is recent; relevance asks whether a process, product, or policy change has made older history stop describing today's reality.
- Ask for the join success rate on every relationship the solution depends on; a 71% join rate means 29% of records arrive carrying none of the features that join was meant to supply.
- Exploratory data analysis is the documented deliverable that closes CPMAI Phase 2 and supplies the evidence for the Task 8 go/no-go.
Overseeing an Analysis You Do Not Perform
Task 7 of Domain III, oversee data evaluation, is where the PMI-CPMAI candidate is tested on a management skill rather than a technical one. You are not expected to run the profiling scripts. You are expected to commission the evaluation, read it critically, ask the questions that expose a weak dataset, and act on what it says — including acting against the project's own momentum. This is the analytical heart of CPMAI Phase 2, Data Understanding, and its output is the evidence base for the Task 8 go/no-go.
Assess Data Quality Dimensions
The acceptance thresholds already exist. You wrote them in Task 1 as numbers with a measurement method against each dimension, before anyone was invested in a result. Task 7 is where those thresholds meet real records, so your questions here are different ones: did somebody actually measure this, was the measurement capable of finding the defect, and what does a failing result cost the project? Insist on a method and a measured value beside every threshold, never an adjective — complete means nothing, income present on 82% of records, 61% in the broker channel means everything.
The table below is a set of probes, not a taxonomy. Read the first column as the question you put to the analyst and the last as the sentence you will have to say to the sponsor if the answer comes back badly.
| Question you put to the analyst | Measurement that answers it | What a failing result looks like on real data | What it costs the project |
|---|---|---|---|
| Are the recorded outcomes actually true? | SME re-check of a stratified sample against source documents or a trusted reference | 9% of records labeled fraud turn out to be chargeback disputes | The model learns a target the business never asked for, tests clean, and the error only surfaces after go-live |
| Where are the values missing — not how many? | Non-null rate per field, broken out by segment, channel and period | Income present on 82% of applications overall but only 61% in the broker channel | Imputation biases every broker prediction, and the aggregate rate hid it from the gate |
| Does the same entity mean the same thing in every system? | Reconcile a sample of entities across systems; compare code lists and encodings | Status is A/I in the CRM, 1/0 in billing, Active/Inactive in the warehouse | Joins fail loudly or mis-join silently; contradictory features cancel each other out in training |
| Is the data fresh enough for the decision the model must support? | Age of the newest record measured against the decision latency written into the requirement | Sensor readings land 36 hours after the event they describe | No model on this feed can meet a 12-hour warning requirement, so the use case itself has to change |
| How many rows would a business rule reject outright? | Percentage of rows passing type, range and format checks | Negative claim amounts, future-dated events, six date formats in one column | Silent parsing errors corrupt features downstream and the model trains on values nobody flagged |
| Is anything being counted twice? | Duplicate-key and near-duplicate detection across extracts and again after joins | The same claim arrives three times from three source systems | Duplicated cases are over-weighted, and the same record straddling train and test inflates every number you report |
Analyze Distributions and Identify Potential Biases or Gaps
Four questions belong to the project manager, not the data scientist:
- Class imbalance. How prevalent is the outcome you are predicting? At 0.3% fraud, a model that predicts never fraud is 99.7% accurate and worthless — which is why raw accuracy is not an acceptable success criterion.
- Outliers. Are extreme values errors to remove, or the rare events the business actually cares about? Deleting them can delete the problem.
- Missingness patterns. Missing values scattered at random are a nuisance you can impute. Missing values concentrated in one channel, region, product, or customer group are missing-not-at-random, and that is a bias risk rather than a gap: the fix will systematically disadvantage exactly the group whose data is absent. Always ask for missingness broken out by segment, never as a single percentage.
- Subgroup representation. Compare the dataset's composition against the deployment population. A model trained on two metro regions and released nationally has a representation problem no algorithm can repair, and it feeds directly into the Domain I bias checks.
Evaluate Data Freshness and Relevance
Freshness asks whether data is recent; relevance asks whether it still describes the world the model will operate in. A process redesign, a repriced product, a policy change, a replaced sensor controller, or a new regulation can make eighteen months of pristine history unrepresentative overnight. Ask directly: what changed in this process during the period covered, and does data from before that change still teach the right lesson? The honest answer sometimes converts three years of history into eight usable months, which changes the sufficiency judgment in Task 8.
Review Data Schema and Structure for Modeling Compatibility
Structural review is unglamorous and catches expensive problems: field types that disagree across sources, categorical fields whose cardinality explodes (40,000 free-text reason codes), join keys that match on only 71% of rows, duplicate records from overlapping extracts, inconsistent units and currencies, mixed encodings, and timestamps in local time with no zone. Ask for the join success rate on every relationship the solution depends on. A 71% join rate is not a 29% shortfall spread thinly: it means 29% of your records arrive carrying none of the features that relationship was supposed to supply, and those records are rarely a random 29%.
Conduct Exploratory Data Analysis
EDA (exploratory data analysis) is the deliverable that closes Phase 2 and feeds the Phase 2 gate. Demand a documented artifact, not a verbal the data looks fine. It should contain, at minimum:
- Row counts by time period and by business segment, against expected volumes.
- Distribution of the target variable and its prevalence, plus how labels were created and by whom.
- Missingness by field and by segment, with patterns called out.
- Duplicate counts, join success rates, and encoding inconsistencies.
- Representation of the dataset against the deployment population.
- Named caveats and open questions, with the analyst's name and the date.
Going Back Is the Methodology Working
CPMAI is iterative by design. When data evaluation shows the recorded target does not exist, the labels cover only escalated cases, or the history predates a process change, the correct move is to return to Phase 1, Business Understanding, and reshape the problem — a different target variable, a narrower persona, a smaller scope. Exam items dramatize this as a setback; the credited response treats it as the methodology doing its job. Discovering the problem here costs a few weeks. Discovering it in Phase 5, Model Evaluation, costs the build.
A data evaluation reports that the income field is missing on 18% of loan applications, and that the missing records are almost entirely applications submitted through the broker channel. The data scientist proposes median imputation and moving on. What should the project manager do?
A predictive maintenance project has three years of sensor history for a production line. Eight months ago the plant replaced the line's controllers and changed the preventive maintenance schedule. The modeling team wants all three years because volume matters. What is the project manager's best action?
The data team closes CPMAI Phase 2 with a single slide: “Data looks good — 2.4 million rows, 96% complete.” The project manager must take the Phase 2 evidence to a go/no-go gate next week. What should the project manager require before accepting the evaluation?