3.1 AWS Glue Jobs, DynamicFrames & PySpark Transformations

3.1 AWS Glue Jobs, DynamicFrames & PySpark Transformations

AWS Glue is a fully managed, serverless data integration service that automates the extraction, transformation, and loading (ETL) of data across enterprise analytics storage environments. At the core of Glue's ETL engine is an optimized, serverless Apache Spark runtime configured with proprietary AWS extensions—most notably DynamicFrames—designed to handle massive, semi-structured, and schema-variant datasets.

Understanding how Glue provisions compute capacity, how DynamicFrames differ from native PySpark DataFrames, and how to apply specialized transformations is critical for passing the AWS Certified Data Engineer - Associate (DEA-C01) exam and building resilient data pipelines.

---

AWS Glue Jobs Execution Engine & Worker Types

AWS Glue decouples data engineers from cluster management by automatically provisioning, configuring, and scaling Apache Spark resources. Glue measures compute capacity in Data Processing Units (DPUs). A single DPU provides a standardized allocation of vCPU and memory compute resources.

Glue Worker Specifications & Sizing Matrix

When configuring a Glue ETL Spark job, data engineers select specific worker types based on memory intensity, compute demands, and shuffle performance requirements:

Exam Tip: Start from measured workload needs. Moving from G.1X to G.2X doubles memory per worker and increases local disk, which can relieve executor pressure, but it does not guarantee an OOM fix. Inspect driver versus executor failure, data skew, shuffle spill, partition sizing, and accidental collection before or alongside scaling.

Execution Classes & Infrastructure Optimizations

1. Standard Execution Class: Ideal for time-sensitive, production workloads that require fast job startup times and guaranteed compute availability.
2. Flex Execution Class: Utilizes non-critical, spare AWS compute capacity for non-urgent ETL jobs (such as overnight batch runs or testing). Flex uses a lower-priced execution class for eligible non-urgent jobs, though startup time can vary with available capacity; compare current Regional rates.
3. Glue Auto Scaling (--enable-auto-scaling=true): Dynamically adds and removes workers based on real-time Spark stage demands. Unlike fixed-size Spark clusters, Glue Auto Scaling monitors executor metrics and releases idle workers during file listing or single-driver write phases, preventing DPU billing waste.
4. S3-optimized output: Current Glue runtimes enable the EMRFS S3-optimized committer for supported Parquet writes by default. Use supported runtime options, partition-aware writes, and compaction; do not invent a connection_options flag.

---

DynamicFrames vs. PySpark DataFrames

While native PySpark DataFrames require a rigid, predefined schema upfront, AWS Glue DynamicFrames maintain an explicit, self-describing dynamic schema that evaluates dataset structures on a row-by-row basis.

Structural & Operational Comparison

Key Scenarios for Using DynamicFrames vs PySpark DataFrames

• Use DynamicFrames when: Ingesting raw JSON, semi-structured logs, or API payloads where fields appear conditionally, nested structures vary across records, or column data types mutate over time. They are convenient for Glue-native transforms and catalog-driven schema mapping, while bookmark support depends on source, API, and transformation context rather than a blanket DynamicFrame requirement.
• Use PySpark DataFrames when: Performing complex analytical window functions (ROW_NUMBER() OVER (...)), multi-table relational JOIN operations, complex statistical UDFs, or interfacing with third-party PySpark libraries.

Converting Between DynamicFrames and DataFrames

Data engineers frequently convert between the two abstractions in a single Glue script to capitalize on the strengths of each API:

# Convert Glue DynamicFrame to PySpark DataFrame
pyspark_df = dynamic_frame.toDF()

# Perform complex PySpark transformations (e.g., Window function)
from pyspark.sql.window import Window
from pyspark.sql.functions import row_number, col

windowSpec = Window.partitionBy("customer_id").orderBy(col("transaction_timestamp").desc())
transformed_df = pyspark_df.withColumn("rank", row_number().over(windowSpec)).filter("rank = 1")

# Convert back to DynamicFrame for optimized AWS Glue catalog writing
transformed_dynamic_frame = DynamicFrame.fromDF(transformed_df, glueContext, "transformed_dynamic_frame")

---

Native DynamicFrame Transformation Methods

AWS Glue provides specialized transformation methods tailored for semi-structured data manipulation:

1. ApplyMapping

Renames columns, alters data types, and drops unwanted fields in a single declarative step:

mapped_dyf = ApplyMapping.apply(
    frame=input_dyf,
    mappings=[
        ("cust_id", "string", "customer_id", "bigint"),
        ("tx_amt", "double", "transaction_amount", "decimal(12,2)"),
        ("raw_payload.event_time", "string", "event_timestamp", "timestamp")
    ],
    transformation_ctx="mapped_dyf"
)

2. ResolveChoice

When a single column contains multiple conflicting data types across different records (e.g., user_id stored as int in early records and string in later records), Glue wraps the column in a Choice type. ResolveChoice resolves these conflicts using one of four strategies:

• make_cols: Creates separate columns for each data type (e.g., user_id_int, user_id_string).
• cast:type: Casts all occurrences to a target type (e.g., cast:string).
• project:type: Retains only records matching a specific type and drops mismatched values.
• make_struct: Combines conflicting types into a nested struct holding both values.

resolved_dyf = ResolveChoice.apply(
    frame=input_dyf,
    choice="make_cols",
    transformation_ctx="resolved_dyf"
)

3. Relationalize

Flattens deeply nested JSON structures containing embedded arrays into a collection of relational DynamicFrames linked by primary and foreign key relationships (root_id), facilitating direct ingestion into relational data warehouses like Amazon Redshift.

# Relationalize produces a DynamicFrameCollection
dfc = input_dyf.relationalize(
    root_name="orders",
    staging_path="s3://my-company-glue-staging/relationalize/"
)

# Extract root table and child array table
orders_root = dfc.select("orders")
orders_items = dfc.select("orders_items")

4. Unnest

Unnests nested struct columns into top-level columns without splitting arrays into separate tables:

unnested_dyf = input_dyf.unnest(transformation_ctx="unnested_dyf")

---

End-to-End AWS Glue PySpark Script Example

The following PySpark ETL script illustrates reading from the Glue Data Catalog, mapping schema fields, converting to a DataFrame for aggregation, converting back to a DynamicFrame, and writing optimized Parquet files to S3:

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from awsglue.dynamicframe import DynamicFrame
from pyspark.sql.functions import col, sum as _sum

# Initialize Glue job arguments and context
args = getResolvedOptions(sys.argv, ['JOB_NAME'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)

# 1. Read from Glue Data Catalog as DynamicFrame
raw_dyf = glueContext.create_dynamic_frame.from_catalog(
    database="sales_db",
    table_name="raw_transactions",
    transformation_ctx="raw_dyf"
)

# 2. Apply Schema Mapping & Type Conversions
mapped_dyf = ApplyMapping.apply(
    frame=raw_dyf,
    mappings=[
        ("store_id", "string", "store_id", "string"),
        ("tx_date", "string", "transaction_date", "string"),
        ("amount", "double", "amount", "double")
    ],
    transformation_ctx="mapped_dyf"
)

# 3. Convert to PySpark DF for Aggregation
df = mapped_dyf.toDF()
aggregated_df = df.groupBy("store_id", "transaction_date") \
                  .agg(_sum("amount").alias("total_daily_sales"))

# 4. Convert back to DynamicFrame
final_dyf = DynamicFrame.fromDF(aggregated_df, glueContext, "final_dyf")

# 5. Write to S3 in Partitioned Parquet format using S3 Direct Writer
glueContext.write_dynamic_frame.from_options(
    frame=final_dyf,
    connection_type="s3",
    connection_options={
        "path": "s3://company-analytics-lake/curated/daily_sales/",
        "partitionKeys": ["transaction_date"]
    },
    format="parquet",
    transformation_ctx="write_dyf"
)

job.commit()