10.2 Dataset Constraint Testing

Constraints as a schema for ML data

Dataset constraint testing asks whether the data adhere to predefined rules. Think of the rule set as a schema for ML data. A database schema names types, nullability, and relationships. An ML constraint set does the same job for files that may never live in a relational table: it is a logical model of what correct data means. If a constraint fails, the data are not ready to train on, or not safe to score in production, even if a scatter plot looks tidy.

Manual spot checks do not scale to the size of modern training dumps or live feature streams. Automate constraint tests in the data pipeline. Route reports to data scientists when training data look wrong, and to operations when live inputs look wrong. The same engine can serve both audiences; the difference is who owns the follow-up.

Single-value constraints

A single-value constraint inspects one attribute on one instance.

Missing. The cell is empty, null, or the attribute is absent from the record. A delay classifier that requires weight_kg must fail a row where weight is blank. Missingness can also mean the column never arrived from an upstream join.

Range. The value must lie in a stated interval. Parcel weight 0.05 kg to 30 kg is a domain range; an 85 kg "parcel" is not a legal input for that product, whether or not it is also a statistical outlier.

Type. The value must match the declared type. If n_stops is specified as an integer, the string "3" or the real 3.7 is a type defect. Integers, strings, and reals are not interchangeable just because a parser can coerce them later.

Multi-value constraints

A multi-value constraint looks across several values, usually one attribute over many instances.

Sum. The total of a column must equal, exceed, or not exceed a limit. The standard teaching example is Formula 1 race points: the total awarded cannot exceed 102 and must exceed 50.5. Worked numbers: a completed race file lists 25, 18, 15, 12, 10, 8, 6, 4, 2, 1, and 1 fastest-lap point. Sum = 102, which satisfies both bounds. If a second 25 is appended (two "winners"), sum = 127, which exceeds 102 and the Sum constraint fires. If a truncated file sums to 40, the total does not exceed 50.5, and Sum fires on the lower bound. Testers apply the same pattern to ML files: daily bonus points in a courier league, allocated minutes in a shift roster, or grouped probabilities that must total near 1.0.

Count. The number of non-null values for an attribute or for instances must equal, exceed, or not exceed a threshold. A batch that requires at least five labelled delayed values fails if only three labels arrived.

Duplicate. Identical or near-identical attribute values or whole instances are limited, often to zero. Two rows with parcel_id P-102 is a Duplicate failure when identifiers must be unique.

Useful. An attribute used as a learning feature should show some repeated values. If every entry is unique, like a raw ID or a high-resolution timestamp, the column typically provides no useful pattern for the model to learn. Duplicate and Useful are easy to mix up on a two-point item. Duplicate asks whether the same ID appeared twice as a data defect. Useful asks whether, even if IDs are perfectly unique, you should feed that column to the model. Unique IDs can pass Duplicate (limit zero duplicates) and still fail Useful as features.

Outlier. Values that sit far from the rest of the distribution are flagged. A domain range check and an outlier check can both fire on the same cell: 85 kg may be outside 0.05–30 and also a statistical outlier in a batch of 1–5 kg parcels.

Comparison constraints

Greater Than. One attribute must exceed another on the same instance. The teaching example: lines of code (LOC) for a program must exceed its count of defective LOC. Module A with loc=500 and defective_loc=40 passes. Module B with loc=80 and defective_loc=80 fails because 80 is not greater than 80. Module C with loc=40 and defective_loc=55 fails as well. The same pattern appears as distance_km must exceed last_mile_km.

Correlate. Values of one attribute must travel with another. The teaching example: every student whose marks sit at least 1.33 standard deviations above the mean also has grade A.

Worked marks: 62, 68, 70, 71, 94. Sum = 365, mean = 73. Deviations: -11, -5, -3, -2, +21. Squared deviations: 121, 25, 9, 4, 441. Sum of squares = 600. Sample standard deviation s = sqrt(600 / 4) = sqrt(150) ≈ 12.25. Then 1.33 × s ≈ 16.29. Threshold = 73 + 16.29 ≈ 89.29. Only the student with 94 clears the threshold. If that student is labelled B, Correlate fires. If labelled A, Correlate is satisfied for this batch. Testers compute the threshold from the dataset under test (or from a documented reference rule) and then check the paired attribute. Do not eyeball "94 looks high" and skip the 1.33 calculation; the exam item is testing that you can apply the rule.

Worked scenario: which constraints fire?

This is the shape of a two-point apply item: you are given a mini-table and a constraint list, and you must mark every rule that fires, not just the first defect you notice.

A pipeline for a parcel-delay classifier declares these rules for a six-row batch:

1. Missing — weight_kg is required.
2. Range — weight_kg in [0.05, 30].
3. Type — n_stops is an integer.
4. Sum — total bonus_points must exceed 50.5 and must not exceed 102.
5. Count — at least five non-null delayed labels.
6. Duplicate — parcel_id unique (zero duplicates allowed).
7. Useful — candidate model features should not be all-unique IDs or timestamps.
8. Outlier — extreme weight_kg relative to the batch.
9. Greater Than — distance_km exceeds last_mile_km.
10. Correlate — any row with priority_score at least 1.33 standard deviations above the batch mean has service_tier = express.

Walk every rule, in order:

• Missing fires on P-103 (blank weight).
• Range fires on P-106 (85.0 is above 30).
• Type fires on P-105 (n_stops is the string "3", not an integer).
• Sum: 18 + 25 + 12 + 20 + 15 + 22 = 112, which exceeds 102, so Sum fires. The lower bound 50.5 is met; the upper bound is not.
• Count does not fire: six delayed labels are present, and the rule asked for at least five.
• Duplicate fires: P-102 appears on two rows.
• Useful fails for parcel_id and scan_ts as features (identifiers and timestamps with no repeated learning pattern). Duplicate can fail on the repeated P-102 at the same time; the two rules still answer different questions.
• Outlier fires on 85.0 kg beside a cluster of roughly 1–4 kg weights.
• Greater Than fires on the first P-102 row: distance 12 is not greater than last_mile 12.
• Correlate: scores 50, 88, 42, 55, 48, 91. Mean = 374 / 6 ≈ 62.33. Sample standard deviation ≈ 21.47. Threshold ≈ 62.33 + 1.33 × 21.47 ≈ 90.9. Only 91 meets the threshold. That row is standard, not express, so Correlate fires.

Exam tactic: list every constraint, mark fire / no-fire, and only then choose the option that matches the full set. Two-point items often hide one clean pass (here, Count) next to several failures so that "everything is broken" is also wrong. Stopping after the blank weight would miss Sum, Duplicate, Type, Greater Than, and Correlate — and would miss most of the marks.

When the same checks run in production, the pipeline should emit a machine-readable report: which rule, which rows, which split (training versus live). Data scientists use training failures to stop a fit. Operations use live failures to quarantine inputs, open an incident, or trigger a fallback policy. Do not treat a red constraint dashboard as a model-accuracy problem until the data rules have been examined.