Section 4.2: Data Quality Policy & Rules

Establishing Data Quality Policy & Requirements

Data quality cannot be achieved through ad-hoc clean-up efforts. It requires a formal framework of policies, requirements gathering, and rule definition. A Data Quality Policy is a formal, high-level document approved by the Data Governance Council that outlines the organization's commitments, standards, roles, and responsibilities regarding data quality. It establishes the mandate for monitoring, remediation, and reporting.

To implement this policy, the organization must first identify Critical Data Elements (CDEs)—data assets that are vital to operational execution, financial reporting, regulatory compliance, or strategic decision-making. Once CDEs are identified, business analysts and data stewards collaborate to define specific data quality requirements.

---

Business Rules vs. Technical Rules

Data quality requirements are articulated through rules. It is critical to distinguish between Business Rules and Technical Rules, as this distinction is a common focus of the CDMP exam.

Business Rules

Business rules are declarative statements that define or constrain some aspect of the business. They express business logic, policies, and operational requirements in plain language, independent of how the data is stored or processed. They are owned by the business.

• Characteristics: Expressed in business terminology, technology-agnostic, and understandable by non-technical stakeholders.
• Example: 'A credit card customer must be at least 18 years of age at the time of application.'

Technical Rules

Technical rules are the system-level implementation and translation of business rules. They are represented as SQL queries, database constraints (e.g., CHECK constraints), ETL (Extract, Transform, Load) validation logic, or regular expressions. They are executed by data quality engines or database systems.

• Characteristics: Written in programming or database syntax, schema-specific, and executable by computers.
• Example: CHECK (DateOfBirth <= DATEADD(year, -18, GETDATE())) or a database trigger enforcing age validation.

---

Data Profiling Techniques

Data Profiling is the process of examining data from an existing source and collecting statistics or informative summaries about that data. It is the foundation of any data quality assessment, allowing data professionals to discover the actual state of data, rather than relying on outdated documentation or assumptions.

DAMA recognizes three main categories of data profiling techniques:

1. Attribute Analysis (Column Profiling)

Attribute analysis focuses on the characteristics of data within a single column. It helps identify basic structural anomalies and data distributions. Common statistics collected during column profiling include:

• Cardinality: The number of unique values in the column. Low cardinality indicates lookup fields (e.g., Gender), whereas high cardinality indicates identifiers (e.g., Customer ID).
• Value Distribution & Frequency: How often each value appears. This highlights anomalies (e.g., an unexpected number of 'NULL' or 'Unknown' entries).
• Data Type & Length: Verifying if the actual data type matches the metadata definition (e.g., checking if a numeric column contains text values).
• Min / Max / Mean / Median: Identifying statistical outliers (e.g., a transaction amount of -$5,000 or an age of 999).

2. Structure Profiling (Format & Pattern Analysis)

Structure profiling evaluates whether the values in a column conform to a specific expected pattern or structural format.

• Pattern Matching: Using regular expressions (Regex) to ensure strings match standard templates.
• Example: Checking if telephone numbers conform to the North American Numbering Plan pattern (###) ###-####. If a value is stored as 123-45-678, it fails the pattern analysis.

3. Relationship Profiling (Cross-Attribute & Referential Analysis)

Relationship profiling examines the connections and dependencies between multiple columns within the same table or across different tables.

• Referential Integrity Analysis: Ensuring that foreign key values in a child table exist as primary keys in the parent table. For example, verifying that every Order.CustomerID points to a valid Customer.CustomerID.
• Functional Dependency Analysis: Checking if the value of one attribute determines the value of another. For example, if the ZIP_Code is '10001', the City must be 'New York'. A violation occurs if the database contains a record with ZIP_Code '10001' and City 'Los Angeles'.
• Redundancy Analysis: Identifying overlapping data across different systems to facilitate integration and master data consolidation.

---

The Data Quality Requirements Gathering Process

Defining data quality requirements is a structured process that bridges the gap between business needs and technical enforcement. The process typically involves five key steps:

1. Identify Stakeholders & Data Consumers: Determine who uses the data, how they use it, and what downstream systems rely on it.
2. Define Critical Data Elements (CDEs): Because resources are limited, organizations cannot monitor every field. They must prioritize CDEs—data points that are critical to operational, financial, or regulatory success.
3. Specify Business Rules: Work with stakeholders to define business expectations (e.g., 'A customer cannot have a shipping address without a corresponding postal code').
4. Establish Data Quality Thresholds: A threshold is the minimum acceptable level of quality for a specific metric. For example, the threshold for completeness on a critical billing field might be set to 99.9%, whereas the threshold for an optional field, like middle_name, might be set to 60%.
5. Translate to Technical Rules & Deploy: Engineers write the code, constraints, or DQ engine configurations to continuously monitor the data against these thresholds, outputting alerts when thresholds are breached.