Outliers, Relationships, Linkages, and Integrity

Outliers, Relationships, Linkages, and Integrity

Quick Answer: Before trusting multi-source analysis, CBDA practitioners detect outliers and noise, validate relationships and linkages across systems, assess dependencies that could make findings fragile, and confirm data integrity through the preparation pipeline. Never auto-delete outliers solely because they are extreme—investigate whether they are errors, rare events, or the very cases the business must understand.

This section closes Domain 2’s “make data usable” arc. Quality dimensions and cleaning get you a table; relationships and integrity determine whether tables mean something together. Outliers test whether your preparation preserved signal or erased it. On scenario exams, strong candidates slow down at joins and anomalies instead of rushing to charts.

Detecting Outliers, Noise, and Notable Features

An outlier is an observation that appears inconsistent with the bulk of the data—extreme values, impossible combinations, or rare patterns. Noise is irrelevant variation that obscures signal. A feature (in the analytics sense) may be a raw field or engineered attribute; some “weird” features are defects, others are predictive gold.

Detection approaches (conceptual)

• Univariate reviews — min/max, percentiles, boxplots, histograms; flag values beyond agreed business ranges (negative ages, 200-hour workdays).
• Multivariate reviews — combinations that should not co-occur (canceled order with shipped quantity > 0; pediatric ward with age 87).
• Time-based spikes — sudden volume or value jumps that may be true campaigns or broken feeds.
• Rule-based checks — business integrity rules (price × quantity ≈ line total within tolerance).
• Model residuals (later) — points the model cannot fit; still require business investigation, not blind deletion.

Profiling remains the workhorse technique for surfacing candidates before formal modeling.

Investigate vs remove: a business-first rule

Exam trap: “Remove all points beyond 1.5× IQR automatically.” Statistical rules are candidate generators, not automatic business truth. For CBDA, connect the action to the research question:

• Studying typical small-business spend → large enterprise outliers may be segmented out of that cohort analysis.
• Studying total revenue risk → the same outliers are essential.
• Building a robust central estimate → sensitivity analyses with and without extremes may be reported.
• Investigating fraud → extremes are the subject, not the contaminant.

Always record counts removed, rules used, and impact on KPIs. Silent outlier deletion is a reproducibility and ethics problem.

Relationships in Data

Business questions almost never live in a single flat file forever. Relationships describe how entities connect: customers place orders; orders contain lines; lines reference products; products roll into categories; tickets link to accounts; claims link to policies.

CBDA practitioners should be fluent in:

• Cardinality — one-to-one, one-to-many, many-to-many. Misunderstanding cardinality causes double-counting (for example, joining customers to tickets without aggregation before averaging “spend per customer”).
• Hierarchical relationships — employee → manager; SKU → family → category; store → district → region. Rollups must use consistent hierarchy versions over time.
• Temporal relationships — as-of joins (customer segment at time of order), slowly changing dimensions, and effective-dated rates.
• Identifier relationships — natural keys vs surrogate keys; email vs customer_id; fuzzy identity resolution across systems.

When a stem says two dashboards disagree, relationship errors (different grains, different join paths) are as likely as pure quality dimension failures.

Joins, Linkages, and Cross-Source Integration

Linkage is the practical act of connecting records across sources so analysis can span systems. Common patterns:

1. Exact key joins — order_id, policy_number, ISBN. Fast and clear when keys are unique and shared.
2. Reference lookups — code tables for status, country, product hierarchy.
3. Identity resolution — matching customers across CRM, billing, and web analytics when no shared key exists (rules, probabilistic matching).
4. Bridge tables — resolving many-to-many (students ↔ courses, patients ↔ providers).
5. Temporal linkage — matching events within time windows (click within 24 hours before purchase).

Join logic changes the population

This is a high-yield exam theme:

• Inner join keeps only matched records → may drop customers without web activity and bias digital engagement studies.
• Left join from the business population keeps unmatched with nulls → preserves population but creates missingness to handle.
• Fan-out joins (one-to-many without aggregation) multiply rows → inflate sums and distort averages.

Before celebrating a merged dataset, re-check: row counts, distinct entity counts, duplicate rates, null rates on key attributes, and reconciliation of control totals (revenue, headcount, claim counts) against source systems.

Linkage quality checks

• Match rate by source and segment (low match rates in one region signal a key problem).
• Orphan rates (orders without customers; lines without headers).
• Conflicting attributes after match (two birth dates—back to consistency).
• Latency differences (warehouse delayed vs operational system live).

Bad linkage is a relationship/integrity failure even when each source scores well alone.

Dependencies and Sensitivity Awareness

Dependencies among variables and process steps affect how stable and interpretable results will be.

• Correlated features — marketing spend and season both rise together; attributing lift to one without care misleads decisions (foreshadows Domain 3 correlation vs causation themes).
• Hierarchical nesting — patients in hospitals, students in classrooms; treating every row as independent can overstate certainty.
• Target leakage dependencies — post-outcome fields that depend on the result.
• Pipeline dependencies — a “clean” field that is computed only when another system is up may be missing exactly when operations are stressed.
• Clustering of errors — one bad extract day can create a cluster of outliers that look like a demand shock.

Sensitivity awareness means asking: if this assumption, join rule, or outlier treatment changes, do recommendations flip? CBDA-aligned practitioners push for simple sensitivity checks (alternate join rules, with/without top 1% spenders, alternate churn definitions) before executives commit resources.

You do not need to implement advanced dependency modeling for every scenario; you need to notice when findings hinge on fragile linkages or unexamined extremes and escalate that risk in plain language.

Data Integrity Assessment

Data integrity is the assurance that data remains whole, accurate, and consistent with business rules as it is created, moved, transformed, and linked. It overlaps quality dimensions but emphasizes end-to-end trustworthiness:

1. At source — capture controls, authorization, audit trails.
2. In transit / extract — complete extracts, checksum or control-total validation, no truncated files.
3. In transformation — deterministic rules, tested edge cases, no silent type coercion disasters.
4. In linkage — stable keys, documented match logic, monitored match rates.
5. In use — consumers cannot casually overwrite certified fields without governance.

Integrity assessment questions for scenarios:

• Do control totals match before and after load?
• Are primary keys unique at the stated grain?
• Do foreign keys resolve to parents?
• Are updates and deletes handled so history remains explainable?
• Can we prove who changed a critical rule and when?

When integrity fails, analysis may still run—and produce confident nonsense. Integrity is therefore a pre-analysis gate alongside dimension scorecards.

Putting It Together: A Scenario Walkthrough

A hospital system wants to analyze 30-day readmissions using EHR discharge data joined to claims. The practitioner:

1. Profiles length-of-stay and charge outliers; investigates a handful of 0-day stays that prove to be registration errors (correct/exclude) and a true long-stay outlier that is clinically real (keep, segment).
2. Maps relationships — encounter → patient → facility; claims may be many per encounter.
3. Chooses linkage — enterprise MRN plus discharge date window; measures match rate by facility; finds one acquired clinic with poor MRN hygiene (integrity/consistency issue).
4. Avoids fan-out — aggregates claims to encounter grain before computing cost metrics.
5. Documents dependencies — readmission flag depends on a specific clinical definition; results are sensitive to whether observation stays count as admissions.
6. Reports integrity limitations — clinic X excluded until identity resolution improves; insights labeled as incomplete for that population.

That is Domain 2 practice at exam depth: technical enough to be correct, business enough to be useful.

Connection Forward to Analysis and Interpretation

Domain 3 (Analyze Data) assumes you can trust structures you built here. Domain 4 (Interpret and Report) assumes caveats about outliers and match rates appear in the narrative. Domain 5 (Influence Decisions) assumes you will not let executives act on a join artifact. Domain 6 (strategy/governance) institutionalizes integrity monitoring so every project is not a one-off fire drill. Learn the vocabulary now; reuse it across the blueprint.

Study Checklist for This Section

• Define outlier vs noise and list investigate-vs-remove decision factors tied to the research question.
• Explain how inner joins and fan-out joins change populations and metrics.
• Describe three linkage patterns and two linkage quality metrics (match rate, orphans).
• Give examples of feature dependency and result sensitivity a BA should flag.
• List integrity checks across source, extract, transform, link, and use.