3.3 Data Preparation & Quality Rules with AWS Glue DataBrew & Data Quality

3.3 Data Preparation & Quality Rules with AWS Glue DataBrew & Data Quality

Ensuring data cleanliness, schema compliance, and statistical validity is essential for enterprise data lakes and downstream machine learning models. AWS Glue provides two complementary tools for data quality management:

1. AWS Glue DataBrew: A visual, interactive data preparation tool designed for analysts and engineers to clean, transform, and profile data without writing code.
2. AWS Glue Data Quality: A programmatic framework powered by open-source Amazon Deequ that enables data engineers to define, monitor, and enforce data quality rulesets using Data Quality Definition Language (DQDL) directly inside Glue ETL pipelines.

---

AWS Glue DataBrew: Visual Data Cleaning & Profiling

AWS Glue DataBrew accelerates data preparation by allowing users to visually inspect datasets, build reusable transformation recipes, and run automated data profiling jobs.

DataBrew Key Concepts & Components

• Datasets: Connections to data stored in Amazon S3, AWS Glue Data Catalog, Amazon Redshift, or Amazon RDS.
• Projects & Interactive Sessions: An interactive workspace where users view sample data records, evaluate column statistics, and visually apply transformations.
• Recipes: An ordered sequence of data transformation instructions (e.g., replace missing values, unpivot columns, apply one-hot encoding, extract regex patterns). Recipes can be published and versioned.
• Recipe Jobs: Automated background batch jobs that apply a published Recipe to entire target datasets in S3 or Redshift.
• Profile Jobs: Batch jobs that evaluate an entire dataset to generate comprehensive statistical quality metrics.

Statistical Metrics Generated by DataBrew Profile Jobs

DataBrew Profile Jobs calculate over 40 detailed statistical indicators across datasets:

---

AWS Glue Data Quality & Data Quality Definition Language (DQDL)

AWS Glue Data Quality allows data engineers to measure and monitor the quality of data stored in S3, Glue Data Catalog tables, or flowing through Glue ETL pipelines. Under the hood, Glue Data Quality leverages Amazon Deequ, an open-source library built on top of Apache Spark that computes data quality metrics at scale.

Authoring DQDL Rulesets

Data quality expectations are declared using Data Quality Definition Language (DQDL). A DQDL document consists of a list of rules evaluating specific attributes of a dataset:

Rules = [
    Completeness "customer_id" >= 0.99,
    IsUnique "transaction_id",
    ColumnLength "zip_code" = 5,
    ColumnValues "account_status" in ["ACTIVE", "SUSPENDED", "CLOSED"],
    RowCount > 10000,
    DistinctValuesCount "country" <= 250,
    CustomSql "SELECT COUNT(*) FROM primary_table WHERE transaction_amount < 0" = 0
]

Core DQDL Rule Types & Syntax Matrix

---

Integrating Data Quality into AWS Glue ETL Pipelines

Glue Data Quality can be executed directly inside PySpark Glue ETL scripts using the EvaluateDataQuality transform. This enables automated decision-making based on rule evaluations.

Actions on Data Quality Failure & Quarantine Pattern

A rule result does not silently enforce a pipeline decision. In Glue Studio, configure the Data Quality transform's outcome action to continue or stop the job at the intended point. In a script, inspect the returned rule or row outcomes and deliberately raise an error, branch, or quarantine rows according to the contract. Publishing options can send evaluation metrics to CloudWatch and detailed results to S3.

Use EvaluateDataQuality().process_rows when row-level routing is required. Its DynamicFrameCollection contains ruleOutcomes and rowLevelOutcomes; the latter includes DataQualityEvaluationResult plus pass, fail, and skip rule arrays. Filter those explicit fields rather than relying on a nonexistent failJob parameter or ConditionalDataQualityResult transform.

Glue PySpark DQDL Integration Example

import sys
from awsglue.context import GlueContext
from awsglue.dynamicframe import DynamicFrame
from awsglue.job import Job
from awsglue.transforms import SelectFromCollection
from awsglue.utils import getResolvedOptions
from awsgluedq.transforms import EvaluateDataQuality
from pyspark.context import SparkContext
from pyspark.sql.functions import col

args = getResolvedOptions(sys.argv, ["JOB_NAME"])
glueContext = GlueContext(SparkContext())
job = Job(glueContext)
job.init(args["JOB_NAME"], args)

input_dyf = glueContext.create_dynamic_frame.from_catalog(
    database="raw_db", table_name="incoming_orders"
)

rules = """
Rules = [
    Completeness "order_id" = 1.0,
    IsUnique "order_id",
    ColumnValues "total_amount" > 0
]
"""

outcomes = EvaluateDataQuality().process_rows(
    frame=input_dyf,
    ruleset=rules,
    publishing_options={
        "dataQualityEvaluationContext": "orders_dq_context",
        "enableDataQualityCloudWatchMetrics": True,
        "enableDataQualityResultsPublishing": True,
        "resultsS3Prefix": "s3://company-dq-reports/orders/",
    },
    additional_options={"performanceTuning.caching": "CACHE_NOTHING"},
)

row_outcomes = SelectFromCollection.apply(
    dfc=outcomes, key="rowLevelOutcomes"
).toDF()
metadata = [
    "DataQualityRulesPass", "DataQualityRulesFail",
    "DataQualityRulesSkip", "DataQualityEvaluationResult",
]
valid_df = row_outcomes.filter(
    col("DataQualityEvaluationResult") == "Passed"
).drop(*metadata)
quarantine_df = row_outcomes.filter(
    col("DataQualityEvaluationResult") == "Failed"
)

glueContext.write_dynamic_frame.from_options(
    frame=DynamicFrame.fromDF(valid_df, glueContext, "valid"),
    connection_type="s3",
    connection_options={"path": "s3://company-analytics-lake/curated/orders/"},
    format="parquet",
)
glueContext.write_dynamic_frame.from_options(
    frame=DynamicFrame.fromDF(quarantine_df, glueContext, "quarantine"),
    connection_type="s3",
    connection_options={"path": "s3://company-quarantine/orders/"},
    format="parquet",
)

job.commit()