4.1 Amazon EMR Cluster Architecture, Spark & Hadoop Ecosystem

4.1 Amazon EMR Cluster Architecture, Spark & Hadoop Ecosystem

Amazon EMR (formerly Elastic MapReduce) is the industry-standard managed big data platform on AWS for processing petabyte-scale datasets using open-source frameworks such as Apache Spark, Apache Hive, Presto/Trino, Apache HBase, and Apache Flink. By decoupling compute resources from data storage, EMR enables data engineering teams to scale compute clusters dynamically while maintaining cost-effective, persistent data lakes on Amazon S3.

Understanding the internal architecture of EMR clusters, node roles, storage abstractions, framework integrations, and cluster lifecycle management is a core requirement for the AWS Certified Data Engineer - Associate (DEA-C01) exam.

---

Amazon EMR Cluster Node Architecture

An Amazon EMR cluster consists of a collection of Amazon EC2 instances grouped into virtual node types. Each node type serves a specific architectural function within the Hadoop YARN (Yet Another Resource Negotiator) cluster topology.

EMR Node Classification & Responsibilities

Exam Tip: If an exam scenario asks how to cut costs for an EMR cluster processing data stored on Amazon S3 without incurring risk of HDFS data loss or HDFS block rebalancing overhead, the optimal architecture is to maintain a minimal set of On-Demand Core nodes and scale compute capacity using Task Nodes on EC2 Spot Instances.

---

Storage Layer Architecture: HDFS vs. EMRFS vs. Amazon S3

Amazon EMR supports two distinct storage abstractions for big data processing workloads: local Hadoop Distributed File System (HDFS) and the EMR File System (EMRFS).

Architectural Comparison: HDFS vs. EMRFS

                               +-----------------------------------+
                               |      Amazon EMR Cluster           |
                               |                                   |
                               |  +--------------+ +------------+  |
                               |  | Primary Node | | Core Nodes |  |
                               |  +--------------+ +------------+  |
                               |                         |         |
                               +-------------------------|---------+
                                                         | EMRFS
                                                         v
                                            +-------------------------+
                                            |      Amazon S3          |
                                            |   (Persistent Lake)     |
                                            +-------------------------+

Advanced EMRFS Capabilities

1. Current S3 processing boundary: S3 Select is closed to new customers. Prefer columnar formats, partition pruning, Spark predicate pushdown, and byte-range reads; existing S3 Select customers can continue using it.
2. KMS Encryption: EMRFS integrates natively with AWS Key Management Service (AWS KMS) to enforce transparent client-side or server-side encryption (SSE-KMS / CSE-KMS) for all data written to S3.
3. Consistent View (Historical): Modern Amazon S3 provides strong read-after-write consistency for all applications out of the box, rendering legacy EMRFS DynamoDB consistency checks obsolete.

---

Big Data Open-Source Ecosystem Frameworks on EMR

Amazon EMR packages optimized builds of popular Apache ecosystem tools. AWS engineers continuously optimize these open-source engines specifically for AWS infrastructure:

Key Ecosystem Engines

• Apache Spark: Distributed, in-memory processing engine for large-scale data processing, batch ETL, streaming, and machine learning. The EMR Runtime for Apache Spark is up to 3.5x faster than standard open-source Spark, reducing execution times and AWS costs.
• Apache Hive: Data warehouse software that facilitates reading, writing, and managing large datasets residing in distributed storage using SQL. EMR Hive can use the AWS Glue Data Catalog as a centralized, external Hive Metastore.
• Presto / Trino: High-performance, distributed SQL query engine designed for fast interactive analytics against heterogeneous data sources (S3, Redshift, MySQL, Cassandra) without data movement.
• Apache Flink: Open-source framework for stateful stream processing over unbounded data streams, supporting event-time processing and exactly-once semantics.
• Apache Hudi / Apache Iceberg / Delta Lake: Open table formats supported on EMR that bring ACID transactions, time travel, and upsert/delete primitives to Amazon S3 data lakes.

---

Cluster Lifecycle: Bootstrap Actions vs. EMR Steps

EMR clusters can be provisioned as Long-Running Clusters (kept active for continuous interactive queries) or Transient Clusters (spin up, execute a specific workload, and terminate immediately upon completion).

1. Bootstrap Actions

Bootstrap actions are custom shell scripts executed on all cluster nodes (Primary, Core, and Task) before Hadoop, Spark, or other application daemons are installed and initialized. They run with root privileges.

Common Use Cases for Bootstrap Actions:

• Installing custom Python libraries (pip install pandas scikit-learn) or system packages (yum install -y libffi-devel).
• Modifying Linux kernel configuration settings or mounting external storage drives.
• Injecting custom security certificates or configuring proxy settings.

2. EMR Steps

Steps are discrete, sequential units of work submitted to an active or transient cluster. A step can execute a Spark application (spark-submit), a Hive script (hive -f script.q), or a custom JAR file.

EMR Step Action-on-Failure Options:

• TERMINATE_CLUSTER: Automatically shuts down the cluster if the step fails (ideal for transient production pipelines).
• CANCEL_AND_WAIT: Cancels all pending steps in the queue and keeps the cluster running for debugging.
• CONTINUE: Ignores the failure and immediately executes the next step in the queue.

---

Programmatic Provisioning: AWS CLI EMR Cluster Definition

The following AWS CLI command demonstrates how to launch a production-grade EMR cluster configured with Glue Catalog integration, custom Bootstrap Actions, and a transient Spark step:

aws emr create-cluster \
    --name "Production-Sales-Aggregation-Cluster" \
    --release-label emr-6.13.0 \
    --applications Name=Spark Name=Hive \
    --use-default-roles \
    --ec2-attributes KeyName=my-ec2-key,SubnetId=subnet-0a1b2c3d4e \
    --instance-groups \
        InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m5.xlarge \
        InstanceGroupType=CORE,InstanceCount=2,InstanceType=r5.xlarge \
        InstanceGroupType=TASK,InstanceCount=4,InstanceType=c5.2xlarge \
    --bootstrap-actions Path=s3://my-company-emr-bootstrap/install-deps.sh,Name="Install Python Dependencies" \
    --steps Type=Spark,Name="Execute Daily Aggregation",ActionOnFailure=TERMINATE_CLUSTER,Args=[--deploy-mode,cluster,--master,yarn,s3://my-company-emr-code/spark_aggregate.py] \
    --configurations '[
        {
          "Classification": "hive-site",
          "Properties": {
            "hive.metastore.client.factory.class": "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveMetastoreFactory"
          }
        },
        {
          "Classification": "spark-hive-site",
          "Properties": {
            "hive.metastore.client.factory.class": "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveMetastoreFactory"
          }
        }
      ]' \
    --auto-terminate