5.3 AI Bias, Fairness & Mitigating Algorithmic Prejudices

5.3 AI Bias, Fairness, and Inclusion

Core principle: Bias is a system property, not merely a bad word in model output. Inspect who defines the problem, whose data is represented, which errors matter, how the product is used, and who can challenge the outcome.

---

Where Bias Enters

Historical bias reflects inequities already present in society or institutions. Representation or sampling bias leaves some populations underrepresented. Measurement bias uses labels or instruments that work differently across groups. Aggregation bias applies one model or overall measure to populations with materially different patterns. Evaluation bias tests on data unlike actual use. Deployment bias uses a system outside the purpose for which it was evaluated.

Proxies can recreate sensitive attributes: postal code, device, school, language, or purchasing history may correlate with protected characteristics. Feedback loops can amplify a decision when model outcomes affect who appears in future data.

Start With the Decision and Harm

Document the intended use, excluded uses, affected people, benefit, error types, and recourse. A text helper for internal drafts has a different risk profile from facial recognition, credit, employment, health, education, or public-benefit decisions. High-impact uses require specialist legal, domain, security, privacy, and ethics review and may be restricted or inappropriate.

Ask who bears false positives and false negatives. “More accurate overall” is incomplete if a small group experiences far more denial, surveillance, or unsafe failure.

Evaluate Disaggregated Performance

Use relevant subgroups and intersections, sufficiently powered samples, confidence intervals or other uncertainty estimates, and representative operating conditions. Inspect data quality and missingness. A four-times error ratio is a warning to investigate, not enough by itself to name the exact bias cause or prescribe mathematical parity.

Common measures include demographic parity, equal opportunity, equalized odds, predictive parity, calibration, and error-rate differences. These answer different questions and can be mutually incompatible when base rates differ. A domain team must justify the chosen measure against the decision, harm, law, and ethical goal. A universal 80% or four-fifths threshold does not certify fairness across products.

Counterfactual and Adversarial Testing

A counterfactual test changes a relevant attribute while holding other factors as stable as practical to see whether an output changes without justification. Adversarial testing probes difficult lighting, language, disability, device, dialect, or edge conditions. For a biometric product, evaluation should use ethically sourced, representative data and qualified protocols; inventing demographic images or labels with a model can add bias rather than solve it.

A model's self-assessment is not independent fairness evidence. Use observed product tests, audit trails, affected-user research, external review where appropriate, and a route to correct outcomes.

Synthetic Personas

Synthetic personas can prompt teams to consider overlooked needs and generate research questions. They can also reproduce stereotypes and create false confidence. Label them synthetic, avoid claiming demographic truth, and never replace consented research with real and diverse people. Compare proposed personas with known evidence and look for missing segments, accessibility needs, non-users, and people harmed by the current workflow.

Accessibility

Accessibility includes perceivable content, operable interfaces, understandable behavior, robust compatibility, cognitive load, language, and assistive technologies. Use the current accessibility standard that applies to the product and organization rather than hard-coding an old version as universal. Automated scanners find only some barriers. Combine them with keyboard and screen-reader testing, contrast and zoom checks, plain-language review, and disabled-user input.

AI can draft alt text, simplify language, or generate test cases. A human must verify meaning in context; an image description can omit identity, emotion, or purpose and automated simplification can distort legal or safety information.

Product Backlog and Definition of Done

Record bias and accessibility risks as transparent Product Backlog work. Add measurable Definition of Done criteria when they are applicable and executable: representative evaluation, disaggregated results, documented limitations, accessibility evidence, monitoring, and recourse. Thresholds should be product-specific and reviewed as populations and use change.

Monitoring and Adaptation

Pre-release testing cannot cover every environment. Monitor outcomes, complaints, overrides, data drift, subgroup performance, and accessibility failures with privacy safeguards. Establish investigation, rollback, correction, and notification. A Retrospective can examine the system and incentives without blaming an individual for surfacing harm.

Participation and Recourse

People affected by a system can reveal harms that a benchmark misses. Provide accessible reporting, correction, appeal, and human support appropriate to the decision. Track whether recourse works in practice, not merely whether a policy link exists. Product discovery should include people most likely to bear errors.