3.1 Distributed Feature Engineering with AWS Glue & DataBrew

Distributed Feature Engineering with AWS Glue & DataBrew

Feature engineering at enterprise scale requires distributed data processing frameworks capable of handling terabytes to petabytes of tabular, semi-structured, and streaming data. For the AWS Certified Machine Learning Engineer — Associate (MLA-C01) exam, you must master how to design scalable, cost-efficient, and maintainable data preparation pipelines using AWS Glue ETL, DynamicFrames, Job Bookmarks, and AWS Glue DataBrew.

+-----------------------------------------------------------------------------------------+
|                        DISTRIBUTED FEATURE ENGINEERING ON AWS                           |
|                                                                                         |
|   [Raw Data Sources]       [Processing Engines]               [Feature Outputs]         |
|   - S3 Raw Lake     --->   - AWS Glue ETL (PySpark / Ray) --> - S3 Cleaned / Curated    |
|   - RDS / DynamoDB         - DynamicFrames Transformations    - SageMaker Training S3   |
|   - Kinesis Streams        - AWS Glue DataBrew (Visual)       - SageMaker Feature Store |
|   - Redshift / JDBC        - PySpark ML Pipelines             - Glue Data Catalog       |
+-----------------------------------------------------------------------------------------+

---

1. AWS Glue ETL Architecture & Job Types

AWS Glue is a serverless data integration service that discovers, prepares, moves, and integrates data from multiple sources for machine learning and analytics. Glue manages the underlying compute infrastructure, provisioning Spark or Ray environments on demand.

Comparison of AWS Glue Job Types

[!TIP] Exam Sizing Rule: If an exam question describes a lightweight feature extraction script (e.g., transforming a 500 MB CSV reference lookup or running basic NumPy/Pandas operations) that runs periodically, selecting a Python Shell job (at 0.0625 or 1 DPU) is significantly more cost-effective than provisioning a distributed PySpark cluster.

---

2. DynamicFrames & Specialized Glue Transformations

In standard Apache Spark, a DataFrame requires an explicit schema where every record in a column conforms to the same data type. However, raw ML data ingested from IoT devices, web event streams, or NoSQL databases often contains schema drift, nested fields, and heterogeneous data types (such as integers mixed with strings in the same column).

AWS Glue introduces the DynamicFrame, an extension of Spark DataFrames where each record is self-describing via a DynamicRecord. DynamicFrames allow schema flexibility without failing on unexpected types.

+-----------------------------------------------------------------------------------------+
|                           DYNAMICFRAME TRANSFORMATION FLOW                              |
|                                                                                         |
|   +-----------------------+     ResolveChoice      +-----------------------+            |
|   | DynamicFrame          | -------------------->  | Clean DynamicFrame    |            |
|   | - colA: int / string  |  (cast:int / make_cols)| - colA: int           |            |
|   +-----------------------+                        +-----------------------+            |
|               |                                                |                        |
|               | Relationalize                                  | toDF()                 |
|               v                                                v                        |
|   +-----------------------+                        +-----------------------+            |
|   | Flattened Tables      |                        | PySpark DataFrame     |            |
|   | - Root Table          |                        | (Spark MLlib Pipeline)|            |
|   | - Nested Child Tables |                        +-----------------------+            |
|   +-----------------------+                                                             |
+-----------------------------------------------------------------------------------------+

Essential DynamicFrame Methods for ML Feature Engineering

1. ResolveChoice

When a column contains mixed types (e.g., "102" and 102), ResolveChoice determines how Glue resolves the conflict:

• cast:target_type: Casts all values to a specified type (e.g., cast:int or cast:double). Invalid values become null.
• make_cols: Splits the ambiguous column into separate typed columns (e.g., col_int and col_string).
• make_struct: Preserves both types inside a nested structure containing all observed variants.
• project:target_type: Keeps only records of the specified type and drops conflicting types.

# Resolving type conflicts in a customer transaction dynamic frame
resolved_df = dynamic_frame.resolveChoice(
    specs=[('transaction_amount', 'cast:double'), ('account_id', 'cast:string')],
    choice='match_catalog',
    database='ml_feature_store',
    table_name='customer_features'
)

2. Relationalize

Extracts deeply nested JSON structures or arrays into a flattened root DynamicFrame and linked auxiliary child DynamicFrames, generating foreign keys to preserve relationships.

# Flattening nested customer profile documents into relational tables
flattened_collection = dynamic_frame.relationalize(
    root_table_name="customer_root",
    staging_path="s3://ml-lake-staging/relationalize/"
)
# Access the flattened root table
customer_root_df = flattened_collection.select('customer_root')

3. Unbox

Unpacks stringified JSON embedded within a column into a structured DynamicFrame column type.

# Unpacking JSON string payload into a struct
unboxed_df = dynamic_frame.unbox(path="payload_json_str", target_type="struct")

4. Converting between DynamicFrame and PySpark DataFrame

To utilize PySpark's extensive pyspark.ml feature engineering library (such as vector assemblers or one-hot encoders), you must convert between DynamicFrames and DataFrames:

# Convert DynamicFrame to Spark DataFrame
spark_df = dynamic_frame.toDF()

# Perform PySpark ML transformations...
# (e.g., StringIndexer, VectorAssembler)

# Convert back to DynamicFrame for optimized S3/Glue Catalog write
from awsglue.dynamicframe import DynamicFrame
final_dynamic_frame = DynamicFrame.fromDF(transformed_spark_df, glueContext, "final_features")

---

3. Incremental Processing with AWS Glue Job Bookmarks

When training ML models continuously on scheduled intervals (e.g., daily or hourly retraining), processing the entire historical dataset repeatedly is computationally wasteful and expensive. AWS Glue Job Bookmarks maintain state information across job runs to ensure that only new or updated data is ingested.

+-----------------------------------------------------------------------------------------+
|                           GLUE JOB BOOKMARK STATE TRACKING                              |
|                                                                                         |
|   Run #1 (08:00):  [File 1 (07:30)]  [File 2 (07:45)]  ---> Processed & Bookmarked      |
|                                                                                         |
|   Run #2 (09:00):  [File 1] (Skipped) [File 2] (Skipped)                                |
|                    [File 3 (08:15)]  [File 4 (08:50)]  ---> Only New Files Processed    |
+-----------------------------------------------------------------------------------------+

Job Bookmark Operational States

• Enable: Glue records the state (S3 object last-modified timestamps, JDBC transaction IDs, or partitions). Subsequent runs process only records added since the previous run.
• Disable: Glue ignores prior states and processes all source data from the beginning on every run.
• Pause: Glue reads the state from a prior run to avoid reprocessing old data, but does not update the bookmark at the conclusion of the current run (ideal for testing pipeline scripts against a fixed new batch).

import sys
from awsglue.utils import getResolvedOptions
from awsglue.context import GlueContext
from pyspark.context import SparkContext
from awsglue.job import Job

args = getResolvedOptions(sys.argv, ['JOB_NAME'])
sc = SparkContext()
glueContext = GlueContext(sc)
job = Job(glueContext)

# Initialize job with bookmark support
job.init(args['JOB_NAME'], args)

# Read incrementally using the bookmark state
source_dynf = glueContext.create_dynamic_frame.from_catalog(
    database="telemetry_db",
    table_name="device_logs",
    transformation_ctx="device_logs_ctx"  # transformation_ctx is required for bookmarks
)

# ... Feature engineering transformations ...

# Commit state to update the bookmark
job.commit()

[!IMPORTANT] Bookmark Requirement: For Job Bookmarks to track state properly in Glue scripts, you must supply a unique transformation_ctx string in source/sink method calls and invoke job.commit() at the end of the script.

---

4. Visual Feature Preparation with AWS Glue DataBrew

AWS Glue DataBrew is a visual data preparation tool that enables data scientists and ML engineers to clean, normalize, and featurize data with an interactive, point-and-click interface featuring over 250 pre-built transformations without writing code.

+-----------------------------------------------------------------------------------------+
|                             GLUE DATABREW WORKFLOW                                      |
|                                                                                         |
|   [Data Sources]             [Interactive Profiling]         [Production Execution]     |
|   - S3 (CSV, Parquet, JSON)  - 250+ Visual Transforms       - DataBrew Recipe Jobs      |
|   - Redshift / RDS           - Automated Data Profile       - Export to S3 / Lake       |
|   - Lake Formation           - Correlation Matrix           - Trigger via EventBridge   |
|   - AppFlow Connectors       - Missing Value Imputation     - SageMaker Pipeline Step   |
+-----------------------------------------------------------------------------------------+

Key Capabilities of Glue DataBrew for ML Engineers

1. Automated Data Profiling: Generates rich statistical distributions, identifies missing value rates, detects numerical outliers, surfaces duplicate rows, and computes feature correlation matrices across numeric features.
2. Visual Transformation Recipes: Transformations are saved as reusable, version-controlled Recipes (JSON/YAML action sequences) containing steps such as:
   • One-hot encoding and label/ordinal encoding.
   • Mathematical scalers: Z-score standard scaling, Min-Max normalization, logarithmic transforms.
   • Text cleaning: Tokenization, stop-word removal, regex pattern replacement, whitespace stripping.
   • Datetime decomposition: Extracting day of week, hour, quarter, is_weekend indicators.
3. Lineage & Versioning: Tracks end-to-end transformation lineage from raw source to curated destination.
4. Production Recipe Execution: DataBrew recipes can be applied across full multi-terabyte datasets by running DataBrew Recipe Jobs, which can be orchestrated as steps within SageMaker Pipelines or triggered by Amazon EventBridge.

---

5. Distributed Feature Engineering with PySpark ML

When writing custom PySpark feature transformations at scale within AWS Glue ETL, ML engineers utilize pyspark.ml.feature estimators and transformers.

from pyspark.ml import Pipeline
from pyspark.ml.feature import (
    StringIndexer,
    OneHotEncoder,
    VectorAssembler,
    StandardScaler,
    Imputer
)

# 1. Impute missing numerical values with median
imputer = Imputer(
    inputCols=["age", "annual_income", "credit_score"],
    outputCols=["age_imputed", "income_imputed", "credit_imputed"]
).setStrategy("median")

# 2. Convert categorical string labels to category indices
string_indexer = StringIndexer(
    inputCols=["employment_type", "education_level"],
    outputCols=["employment_index", "education_index"],
    handleInvalid="keep"  # Handle unseen categories gracefully in test data
)

# 3. One-hot encode indexed categorical columns
one_hot_encoder = OneHotEncoder(
    inputCols=["employment_index", "education_index"],
    outputCols=["employment_vec", "education_vec"]
)

# 4. Assemble numeric features into a single vector for scaling
numeric_assembler = VectorAssembler(
    inputCols=["age_imputed", "income_imputed", "credit_imputed"],
    outputCol="numeric_features_raw"
)

# 5. Standard scale numeric features to zero mean and unit variance
scaler = StandardScaler(
    inputCol="numeric_features_raw",
    outputCol="numeric_features_scaled",
    withMean=True,
    withStd=True
)

# 6. Assemble final master feature vector for ML model training
final_assembler = VectorAssembler(
    inputCols=["numeric_features_scaled", "employment_vec", "education_vec"],
    outputCol="features"
)

# Combine all transformation stages into a single deterministic Pipeline
feature_pipeline = Pipeline(stages=[
    imputer,
    string_indexer,
    one_hot_encoder,
    numeric_assembler,
    scaler,
    final_assembler
])

# Fit on training data and transform
pipeline_model = feature_pipeline.fit(spark_df)
transformed_dataset = pipeline_model.transform(spark_df)

[!WARNING] StringIndexer Trap: In production streaming or batch test sets, new unseen category strings will cause StringIndexer to throw an error by default (handleInvalid="error"). Always specify handleInvalid="keep" or handleInvalid="skip" in ML feature engineering pipelines to prevent pipeline crashes on new categorical values.

---

6. Performance Tuning & DPU Sizing for AWS Glue ETL

In AWS Glue, compute capacity is measured in Data Processing Units (DPUs). 1 DPU provides 4 vCPUs and 16 GB of memory.

Glue Worker Types Breakdown

+-----------------------------------------------------------------------------------------+
|                             GLUE WORKER TYPES MATRIX                                    |
|                                                                                         |
|   Worker Type   DPU / Worker   vCPU    RAM      Executors / Worker    Best For          |
|   -----------   ------------   ----   ------    ------------------    ---------------   |
|   G.1X          1 DPU          4      16 GB     1 executor            Memory-intensive  |
|   G.2X          2 DPU          8      32 GB     1 large executor      Compute-heavy ML  |
|   G.4X          4 DPU          16     64 GB     1 massive executor    Extreme memory    |
|   G.8X          8 DPU          32     128 GB    1 ultra executor      Terabyte joins    |
|   G.025X        0.25 DPU       2      4 GB      Micro-executor        Streaming logs    |
+-----------------------------------------------------------------------------------------+

Glue Performance Optimization Strategies

1. Glue Auto Scaling: Set --enable-auto-scaling=true. Glue automatically increases or decreases worker nodes based on real-time Spark execution stage metrics, preventing over-provisioning during simple read/write steps and scaling up during heavy transformations.
2. Mitigating Skew with Partitioning: Uneven partition sizes lead to "straggler tasks" where one Spark executor processes significantly more data than others. Use df.repartition(num_partitions, "partition_col") to distribute workloads evenly, or coalesce() to reduce partition count without full shuffling when saving output.
3. Broadcast Joins: When joining a large multi-million row transaction table with a small categorical lookup table (< 100 MB), force a broadcast join using from pyspark.sql.functions import broadcast; large_df.join(broadcast(small_df), "id"). This eliminates expensive distributed shuffle operations across the cluster.
4. DynamicFrame Pushdown Predicates: Filter partitions directly at the catalog level before reading files from S3 to minimize I/O overhead:

   glueContext.create_dynamic_frame.from_catalog(
       database="ml_lake",
       table_name="transactions",
       push_down_predicate="year == '2026' and month == '08'"
   )