Conduct Bias Checks on Model, Data, and Algorithm
Key Takeaways
- The four-fifths rule is the one number to memorize: divide the lowest group's selection rate by the highest group's, and a ratio below 0.80 creates a presumption of adverse impact under EEOC guidance (15% vs 25% gives 0.60 and fails).
- A single headline accuracy figure hides subgroup failure; require disaggregated performance by subgroup, with sample sizes shown, as a written acceptance criterion.
- Demographic parity (equal selection rates) and equalized odds (equal true-positive and false-positive rates) generally cannot both be satisfied when base rates differ, so the governing definition is a documented business and legal choice.
- Mitigation happens at three points: pre-processing in Phase 3 Data Preparation, in-processing in Phase 4 Model Development, and post-processing thresholds in Phase 5 Model Evaluation.
- Deleting a protected attribute from the feature list does not remove bias, because proxies persist — and it destroys your ability to measure subgroup performance at all.
Bias Is a Scheduled Check, Not an Accusation
ECO Domain I, Task 3 asks you to conduct bias checks on the model, the data and the algorithm — three places, not one. Bias in an AI system is rarely someone's prejudice; it is usually a property the data inherited and the training process amplified. Treat it as a defect class you plan for, with owners, dates and pass criteria, exactly as you plan performance testing.
Five Sources of Bias
Historical bias — the data faithfully records past decisions that were themselves unfair. A promotion-recommendation model trained on ten years of internal decisions learns whatever those decisions encoded. The data is accurate; the world it describes was not.
Representation bias — the training sample under-covers part of the deployment population. A predictive-maintenance model built on three plants underperforms at the fourth; a voice interface trained mostly on one accent group fails another.
Measurement bias — the chosen label or feature is a flawed proxy for what you actually care about. Arrest records as a proxy for crime measure enforcement patterns. Historical healthcare spending as a proxy for health need measures access to care.
Aggregation bias — one model is forced onto a genuinely heterogeneous population, so optimization follows the majority segment while a minority segment degrades. One demand forecast covering flagship stores and rural franchises is the everyday version.
Evaluation bias — the test set does not reflect deployment, so the headline metric flatters the model. If the test set holds eleven records for a segment, that segment's reported accuracy means nothing.
Stop Reading One Accuracy Number
A single headline accuracy figure is the most common way subgroup failure slips through a go/no-go gate. Require disaggregated evaluation: the same metrics reported separately for every subgroup that matters, with the sample size beside each so thin cells are visible rather than averaged away.
That carries a consequence you must handle in Phase 2 Data Understanding. To measure performance by subgroup, the attribute has to exist somewhere you can join to. You generally will not use protected attributes as model features, yet you still need them — held separately, access-controlled and governed — to test fairness at all. Decide that with privacy and legal in Phase 2, not in the week before evaluation.
And note that dropping a protected attribute from the feature list does not remove bias. ZIP code, school, device type, employment gaps and purchase history act as proxies, and the model rediscovers the pattern without ever seeing the label.
The One Number Worth Memorizing: The Four-Fifths Rule
Compute the selection rate — the share of each group receiving the favorable outcome — then divide the lowest group's rate by the highest group's rate. Under the U.S. EEOC Uniform Guidelines on Employee Selection Procedures, a ratio below 0.80 (80%, the four-fifths rule) creates a presumption of adverse impact.
Worked example: a screening model advances 25% of Group A and 15% of Group B. 15 divided by 25 is 0.60. That is below 0.80, so it is an adverse-impact indicator, and the Phase 5 go/no-go must not clear until root cause and mitigation are on the table.
Two cautions. Passing 0.80 is a screening threshold, not a safe harbor and not proof of fairness. And it originates in U.S. employment law and is applied more broadly by analogy — know it as the number the exam expects you to compute and act on, then bring counsel for your jurisdiction and use case.
Fairness Definitions in Plain English
Demographic parity means each group receives the favorable outcome at the same rate, regardless of underlying outcomes. Equalized odds means the model is equally right and equally wrong for each group — the same true-positive rate and the same false-positive rate. A third family, predictive parity, asks that a positive prediction be equally trustworthy whichever group it lands on.
Here is the part managers must internalize: when the underlying base rates differ between groups, these definitions conflict mathematically. You cannot satisfy them all at once, and improving one degrades another. There is no neutral default. Choosing which definition governs your project is a business, legal and ethical decision — made explicitly with counsel and the sponsor, recorded in the model card, and defensible later. A team that quietly adopts whichever metric its library defaults to has made a governance decision it had no authority to make.
Mitigation at Three Points
| Point | CPMAI phase | What happens | Manager's note |
|---|---|---|---|
| Pre-processing | Phase 3 Data Preparation | Rebalance or reweight the sample, collect more data for under-covered groups, fix or replace bad labels | Cheapest and most durable, because it addresses the cause |
| In-processing | Phase 4 Model Development | Train with a fairness constraint alongside the accuracy objective | Requires the fairness definition to be chosen first |
| Post-processing | Phase 5 Model Evaluation | Adjust decision thresholds after scoring | Fast, but group-specific thresholds raise legal questions — involve counsel |
Where Bias Enters and How You Catch It
| Bias type | Where it enters | How you detect it | Mitigation |
|---|---|---|---|
| Historical | Phase 2 — labels record past unfair decisions | Compare outcome rates by group against an external benchmark; ask SMEs how those decisions were made | Re-label, change the target, or escalate; no algorithm repairs an unfair label |
| Representation | Phase 2-3 — the sample under-covers a group | Compare dataset composition against the deployment population | Targeted collection, stratified sampling, reweighting |
| Measurement | Phase 1-2 — the proxy label is not the real target | Ask what the column actually measures; validate the proxy against ground truth on a sample | Change the target variable or add corrective features |
| Aggregation | Phase 4 — one model over mixed segments | Disaggregated metrics show one segment far worse than the average | Segment-specific models or interaction features |
| Evaluation | Phase 5 — the test set omits subgroups | Check per-subgroup counts before trusting any metric | Rebuild the test set with minimum per-group sample sizes |
Your Job as the Project Manager
Three commitments cover the enablers.
- Write it into acceptance criteria. "The evaluation report must present performance disaggregated by the defined subgroups, and no subgroup selection-rate ratio may fall below 0.80 without documented business justification" is a scope statement line, not an aspiration.
- Schedule the checks. Phase 2 tests representativeness against the deployment population, Phase 4 checks intermediate results while the model is being developed, and Phase 5 runs the formal fairness test that feeds the go/no-go decision.
- Keep them running in production. Populations shift, new segments arrive, and retraining reintroduces old patterns. Put bias metrics in the same monitoring pack as accuracy and drift, with thresholds, alerts and a named owner. One-and-done fairness testing is precisely the trap this exam punishes.
At the Phase 5 go/no-go review for a resume-screening model, the evaluation report shows overall accuracy of 91%, an advance rate of 34% for one demographic group and 22% for another. What should the project manager do?
A team assures you that its lending model cannot be biased because gender and ethnicity were deleted from the dataset before training began. What is the most accurate assessment of that claim?
A fraud-detection model passed its fairness testing at launch and has been running for nine months. The customer base has shifted toward a new market segment and the model has been retrained twice on recent data. What should the project manager ensure?