9.2 Testing for Bias

Bias in a machine learning system is not a casual synonym for the model made a mistake. Bias, as CT-AI uses the word, means non-random, unfair differences in treatment based on sensitive attributes such as gender, age, or race. Those differences are often unlawful, and they make the system discriminatory even when nobody on the team typed a prejudiced rule. ISTQB CT-AI version 2.0 learning objective AI-5.1.2 (K2) asks you to explain how to test for that kind of bias. The skill is diagnostic. You must know where unfairness can enter, which reviews and analyses can expose it, and how disparate impact analysis uses counterfactuals without fooling yourself.

Two source families: data bias and algorithmic bias

Testing for bias starts by understanding sources. They fall into two families that exam stems love to mix together.

Data bias lives in the examples. The training set may be unrepresentative of the people who will be scored. It may carry historical skew — years of human decisions that already treated groups differently, now frozen as ground truth. It may have been poisoned so that a group is associated with a worse label. In all of those cases the algorithm may be doing exactly what it was asked: reproduce the data. A hiring model trained on ten years of promotions that favored one gender is not discovering talent. It is photocopying a workplace.

Algorithmic bias lives in the algorithm, the model, or the development framework. A common picture is a credit-score decision threshold in a loan system. The score itself might be a compressed history; the threshold is a choice about who is good enough. If that choice — or a similar hyperparameter, a loss weight, or a post-processing rule — treats groups differently even when the relevant non-sensitive evidence is comparable, the unfairness is not only in the table of examples. It is in how the system decides. A threshold that looks neutral on a slide can still cut a group out of credit if that group's scores pile up just below the line for reasons the business has not justified.

You do not have to win a philosophy argument about origin on every exam item. You do have to name a test that would reveal the unfair treatment, wherever it was born. Outcome differences can be produced by dirty data, by a clever-looking rule, or by both at once.

Reviews of the workflow, especially data preparation

Start with reviews of the overall ML workflow, with extra time on data preparation. Preparation is where well-meaning teams introduce bias they will later blame on the model. Filtering noisy rows may drop a language, a clinic, or an age band. Imputing missing income with a global mean may erase a neighborhood. Upsampling one class may clone majority faces until a camera almost never sees anyone else. A review asks which rows were kept, which were dropped, which were synthesized, and who had the power to decide. If the workflow cannot answer those questions, later statistical tests have nowhere solid to stand. Review is not a soft substitute for measurement. It is how you find the step that manufactured the skew.

Dataset documentation reviews

Next, review dataset documentation with unfairness in mind. Useful documentation is not a marketing paragraph. It says how the data were collected, how they were annotated, and which populations are represented — and, just as important, which populations are thin or absent. A face corpus collected at one university campus is not people in general. A claims extract from one insurer's urban book is not all drivers. Annotation notes should say who labelled, under which guide, and what they were told to do with ambiguous cases. Documentation reviews do not replace measurement, but they tell you where to measure and which claims of representativeness are theater. On a K2 item, a stem that lingers on missing collection notes or unexplained annotation rules is pointing you here before it points you at a significance test.

Static analysis of preparation programs and model code

Static analysis of data-preparation programs and of the model implementation looks for anti-patterns and for mishandling of sensitive attributes. Examples include hard-coded filters on sex or age that were meant to be temporary, target leakage from a protected-class proxy, different imputation logic by group, or a threshold applied only after a demographic flag is read. Static analysis will not catch every statistical disparity. It does catch the embarrassing class of defects where the code literally branches on a sensitive field, or drops that field so clumsily that a proxy still carries the same information. Read this as white-box bias testing: you are inspecting how attributes are handled, not waiting for a dashboard to light up.

Exploratory data analysis across attributes

Exploratory data analysis (EDA) of the training data uses visualization and clustering to reveal imbalance, skewed distributions, or anomalous groupings across sensitive attributes. Histograms split by age band, scatter plots colored by gender, and cluster methods that suddenly isolate a racial group are not decorations for a slide. They are tests. If one cluster is almost entirely one group and almost entirely one label, you have a hypothesis: the model may learn the group instead of the legitimate signal. EDA is cheap compared with a production incident, and it works before you spend a training budget. When a question mentions plots, clusters, or odd groupings by gender, age, or race, it is describing this approach, not disparate impact analysis.

Dynamic testing with a known unbiased, representative set

Dynamic testing detects bias in the fitted model even when you cannot see it in the training file. Feed a known unbiased and representative dataset through the system and analyze predictions for statistically significant differences in outcomes across sensitive groups. This is outcome testing. It does not require you to prove whether the poison was in the data or in the code. If comparable people receive different decisions after you have controlled the legitimate factors the business claims to use, the model is treating groups differently. Comparable is the hard word in practice. The exam still expects you to know the shape of the test: a trusted, representative probe set, scored, then compared across groups with a statistical lens rather than a single anecdote.

Label correctness testing as a bias detector

Label correctness testing belongs in the bias toolbox because mislabels can teach a false link between an attribute and an outcome. If annotators systematically tag women's resumes as unqualified for the same evidence, or if accident photos from one neighborhood are labelled fraud at a higher rate, the model will launder that error into a score. Finding and repairing those labels is not a courtesy to the dataset. It is a bias control. When a stem says the unfairness started with how outcomes were tagged, reach for label correctness testing alongside, not instead of, the other approaches.

Disparate impact analysis, step by step

Disparate impact analysis is the structured experiment CT-AI wants you to be able to explain. Walk it in order, because exam options often shuffle the steps.

1. Identify sensitive attributes for the model. The list is not whatever is in the ethics poster. It is the attributes for which unfair differences would be unacceptable or unlawful in this application — often gender, age, and race, sometimes with additional context-specific fields the domain treats as protected.
2. Generate counterfactuals for those attributes. A counterfactual is a twin record that keeps the legitimate evidence as constant as the task allows and changes the sensitive attribute. The classic picture is a loan application that is identical except that gender is swapped from one value to another.
3. Obtain model outputs by presenting those counterfactuals to the model. You are not arguing with the training set here. You are querying the decision procedure as it will actually score cases.
4. Analyze multiple tests for statistical significance to determine whether changing the attribute causes results to change. A single swapped row is an anecdote. A planned set of swaps, scored with an appropriate significance test, is evidence that the attribute moved the outcome.

You may combine attributes. Hidden bias often lives in intersections: age and gender, race and postcode, disability and language. Intersection tests are still disparate impact analysis. They simply treat the combination as the factor you vary. A model that looks fair on gender alone and fair on age alone can still fail when older women are the group that never receives the product.

Take care that counterfactuals are not unrealistic. If you generate a record the model should reject because it is physically or administratively impossible — a pediatric dosage on an adult body record, a license type that cannot exist with that age, a pregnancy flag on a combination the system treats as invalid — the model may respond to the impossibility rather than to bias. That rejection is not evidence that the sensitive attribute was ignored fairly. It is evidence that you asked a silly question. Design twins that a domain expert would still call a plausible application. The warning is practical: bad counterfactuals produce both false alarms and false comfort.

Putting the approaches together

Bias testing is layered on purpose. Reviews and documentation tell you where to look. Static analysis catches explicit mishandling. EDA shows imbalance before training locks it in. Dynamic testing and disparate impact analysis interrogate the model's behavior. Label correctness testing attacks a common teacher of false associations. On a K2 item, match the approach to the clue. A sloppy collection write-up points at documentation review. A threshold on a credit score points at algorithmic bias plus outcome tests. A gender swap on a loan application points at counterfactual disparate impact analysis. Unrealistic twins that the model refuses are a trap, not a proof of fairness.