8.1 Cloud Dataproc: Architecture, Ephemeral vs Persistent Clusters, and Spark Tuning

8.1 Cloud Dataproc: Architecture, Ephemeral vs Persistent Clusters, and Spark Tuning

Exam Focus: The Google Cloud Professional Data Engineer exam rigorously tests your ability to modernize legacy Apache Hadoop and Apache Spark estates on Google Cloud. You must master the architectural decoupling of compute and storage using the Google Cloud Storage connector (gs://), distinguish between persistent and ephemeral (job-scoped) cluster topologies, leverage secondary workers (Spot VMs) for cost-optimized stateless compute, manage schemas with Dataproc Metastore (DPMS), and tune Spark memory configurations (spark.executor.memoryOverhead, spark.memory.fraction, and partition sizing) to resolve out-of-memory crashes.

Enterprise big data platforms historically relied on physical, monolithic Hadoop clusters where distributed storage (HDFS) and distributed compute (MapReduce, YARN, Spark) were tightly coupled to the same bare-metal server chassis. Google Cloud Dataproc transforms this paradigm into a fully managed, cloud-native service that provisions Apache Spark, Apache Hadoop, Apache Hive, and Presto/Trino environments on Google Compute Engine virtual machines in approximately 60 to 90 seconds. Understanding how Dataproc separates storage from compute and orchestrates worker nodes is critical for passing the exam and building cost-effective data pipelines.

---

1. Dataproc Cluster Architecture: Node Roles and Topologies

A Cloud Dataproc cluster is provisioned inside a customer's Virtual Private Cloud (VPC) network and comprises three distinct node categories:

+-----------------------------------------------------------------------------------+
|                         DATAPROC CLUSTER ARCHITECTURE                             |
+-----------------------------------------------------------------------------------+
|                                                                                   |
|  +─────────────────────────────────────────────────────────────────────────────+  |
|  |                        MASTER NODE (Standard VM)                            |  |
|  |  - YARN ResourceManager       - HDFS NameNode        - Spark Driver / Master |  |
|  |  - HiveServer2 / Metastore    - ZooKeeper (HA mode)  - Cloud Logging Agent  |  |
|  +─────────────────────────────────────────────────────────────────────────────+  |
|                                          │                                        |
|            ┌─────────────────────────────┴─────────────────────────────┐          |
|            ▼                                                           ▼          |
|  +──────────────────────────────────+         +──────────────────────────────────+|
|  |   PRIMARY WORKERS (Standard VMs) |         |  SECONDARY WORKERS (Spot VMs)    ||
|  |  - YARN NodeManager (Compute)    |         |  - YARN NodeManager ONLY         ||
|  |  - HDFS DataNode (Storage/State) |         |  - Purely Stateless Compute      ||
|  |  - Persistent Boot & Data Disks  |         |  - No HDFS DataNode Daemons      ||
|  +──────────────────────────────────+         +──────────────────────────────────+|
|            │                                                           │          |
|            └─────────────────────────────┬─────────────────────────────┘          |
|                                          ▼                                        |
|  +─────────────────────────────────────────────────────────────────────────────+  |
|  |           PERSISTENT STORAGE: GOOGLE CLOUD STORAGE (gs://bucket)            |  |
|  |  - Infinite Scale, 99.999999999% Durability, Zero Replication Cost Multiplier|  |
|  +─────────────────────────────────────────────────────────────────────────────+  |
+-----------------------------------------------------------------------------------+

Master Nodes

• Role: Coordinates distributed resource allocation, tracks job execution state, and manages metadata daemons.
• Daemons: Runs the Apache Hadoop YARN ResourceManager, HDFS NameNode, Apache Spark Master, and management agents.
• Single-Node Mode: A single Compute Engine VM hosts both master and worker processes. Used strictly for low-cost software development, prototyping, and testing (no high availability, no horizontal scaling).
• Standard Mode: Exactly one master node paired with two or more worker nodes.
• High Availability (HA) Mode: Provisions three master nodes running an active-passive HDFS NameNode pair with Apache ZooKeeper distributed consensus and quorum journals. If the primary master VM experiences hardware degradation or zone-level disruption, failover occurs transparently without aborting in-flight YARN applications.

Primary Worker Nodes

• Role: Executes distributed data processing tasks and, when HDFS is utilized, hosts persistent distributed blocks.
• Daemons: Runs the YARN NodeManager (allocating CPU vCores and RAM containers) and HDFS DataNode.
• Infrastructure: Provisioned strictly on standard, non-preemptible Compute Engine VM instances. A minimum of two primary workers is required in standard and HA clusters to satisfy minimum HDFS replica constraints.
• Lifecycle: Must remain running throughout the active lifespan of the cluster to ensure cluster stability and prevent HDFS under-replication.

Secondary Worker Nodes (Stateless Spot / Preemptible VMs)

• Role: Provides dynamic, low-cost compute burst capacity for batch processing.
• Daemons: Runs only the YARN NodeManager. Secondary workers never run HDFS DataNodes.
• Infrastructure: Typically provisioned on Compute Engine Spot VMs (or legacy Preemptible VMs), providing a 60% to 91% discount compared to on-demand pricing.
• Stateless Resilience: Because secondary workers do not host HDFS blocks, a sudden preemption by Google Cloud does not cause HDFS block corruption, split-brain scenarios, or filesystem degradation. YARN simply detects the loss of the NodeManager heartbeat and automatically reschedules the interrupted task attempts on remaining active workers.

---

2. Decoupling Compute and Storage: The Cloud Storage Connector

In traditional on-premises deployments, scaling data storage necessitated purchasing physical servers with attached disks, inadvertently paying for unused CPU and RAM. Conversely, scaling compute capacity required provisioning servers whose storage sat idle.

+─────────────────────────────────────────────────────────────────────────────────+
|                     ON-PREMISES COUPLED ARCHITECTURE                            |
|  [ Compute Node ] ──> Attached Disk (HDFS with 3x Replication Factor)           |
|  * Cluster must run 24/7 or data is destroyed. High CapEx and idle waste.       |
+─────────────────────────────────────────────────────────────────────────────────+
                                         │
                                         ▼ Decoupled Modern Cloud Architecture
+─────────────────────────────────────────────────────────────────────────────────+
|                       DATAPROC DECOUPLED ARCHITECTURE                           |
|  [ Dataproc Workers ] ──(GCS Connector / gRPC)──> [ Google Cloud Storage ]      |
|  * Workers are purely stateless compute. Shut down clusters when jobs finish.   |
|  * Storage is paid per GB/month; no 3x replication storage multiplier penalty.  |
+─────────────────────────────────────────────────────────────────────────────────+

The Cloud Storage Connector (gs://)

Dataproc pre-installs the open-source Google Cloud Storage (GCS) connector, an implementation of the Hadoop FileSystem abstraction (fs.gs.impl = com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem). This connector allows Spark, MapReduce, Hive, and Pig to read and write directly to Cloud Storage buckets using standard gs:// URI syntax without code modifications.

Key Architectural Advantages of GCS over HDFS

1. Massive Cost Reduction: HDFS requires a default replication factor of 3 to survive individual hard drive failures, tripling storage capacity requirements (e.g., storing 100 TB of raw data consumes 300 TB of physical disk). Cloud Storage provides 99.999999999% (11 9's) annual durability natively using erasure coding across multi-zone facilities, charging only for the logical bytes stored.
2. Ephemeral Lifecycles: When data resides in Cloud Storage, Dataproc clusters become completely disposable. Clusters can be spun up for a 15-minute Spark transformation and deleted immediately upon completion, driving compute idle costs to zero.
3. Global Data Accessibility: Data written to gs://my-lake/curated/ is instantly readable by other GCP services—such as BigQuery (via external tables or BigLake), Cloud Dataflow, and Vertex AI—without requiring complex network exports or data duplication.
4. Strong Global Consistency: Since late 2020, Cloud Storage provides strong global read-after-write consistency for all object creation, updates, and bucket listings. A file written by Spark stage $N$ is guaranteed to be visible to Spark stage $N+1$.

Exam Trap: Do not store persistent pipeline inputs or final analytical datasets in Dataproc HDFS. Local HDFS should be treated solely as a temporary scratch disk for intermediate shuffle spills or fast micro-caching. All source inputs and production outputs must target gs:// paths.

---

3. Ephemeral (Job-Scoped) vs. Persistent Long-Running Clusters

A fundamental architectural decision tested on the exam is choosing between persistent multi-tenant clusters and ephemeral (job-scoped) clusters.

The Cloud Composer Ephemeral Orchestration Pattern

In enterprise production environments, ephemeral Dataproc clusters are orchestrated using Cloud Composer (managed Apache Airflow) through a three-stage DAG lifecycle:

# Enterprise Ephemeral Dataproc Lifecycle in Cloud Composer / Apache Airflow
from airflow import models
from airflow.providers.google.cloud.operators.dataproc import (
    DataprocCreateClusterOperator,
    DataprocSubmitJobOperator,
    DataprocDeleteClusterOperator,
)
from airflow.utils.trigger_rule import TriggerRule

with models.DAG("ephemeral_spark_pipeline", schedule_interval="@daily") as dag:
    
    # 1. Provision Ephemeral Cluster tailored to the job
    create_cluster = DataprocCreateClusterOperator(
        task_id="create_dataproc_cluster",
        cluster_name="ephemeral-spark-{{ ds_nodash }}",
        region="us-central1",
        cluster_config={
            "master_config": {"num_instances": 1, "machine_type_uri": "n2-standard-4"},
            "worker_config": {"num_instances": 2, "machine_type_uri": "n2-highmem-8"},
            "secondary_worker_config": {
                "num_instances": 10,
                "machine_type_uri": "n2-highmem-8",
                "is_preemptible": True,  # Leverages Spot VMs for burst compute
            },
        },
    )

    # 2. Submit the Spark batch workload
    submit_job = DataprocSubmitJobOperator(
        task_id="submit_spark_job",
        region="us-central1",
        job={
            "reference": {"project_id": "my-gcp-project"},
            "placement": {"cluster_name": "ephemeral-spark-{{ ds_nodash }}"},
            "pyspark_job": {"main_python_file_uri": "gs://my-bucket/scripts/etl.py"},
        },
    )

    # 3. Always tear down cluster, even if the Spark job fails
    delete_cluster = DataprocDeleteClusterOperator(
        task_id="delete_dataproc_cluster",
        region="us-central1",
        cluster_name="ephemeral-spark-{{ ds_nodash }}",
        trigger_rule=TriggerRule.ALL_DONE,  # GUARANTEES TEARDOWN ON FAILURE
    )

    create_cluster >> submit_job >> delete_cluster

Exam Warning: Always ensure the cluster deletion operator specifies TriggerRule.ALL_DONE. If configured with the default ALL_SUCCESS, a failed Spark job will bypass the deletion operator, leaving expensive Compute Engine VMs running indefinitely.

---

4. Primary Workers vs. Secondary Workers (Spot VMs)

Dataproc allows combining regular VMs with Spot VMs to minimize operational expenses:

+─────────────────────────────────────────────────────────────────────────────+
|                      DATAPROC WORKER NODE COMPARISON                        |
+─────────────────────────────────────────────────────────────────────────────+
|  PRIMARY WORKERS (Standard VMs)        SECONDARY WORKERS (Spot VMs)         |
|  - Runs: YARN NodeManager              - Runs: YARN NodeManager ONLY        |
|  - Runs: HDFS DataNode                 - NO HDFS DataNodes                  |
|  - Stateful: Hosts HDFS blocks         - Purely Stateless: Compute worker   |
|  - Never preempted by GCP              - Can be reclaimed at any time       |
|  - Required minimum: 2 instances       - Can scale from 0 to hundreds       |
|  - Full standard VM price              - 60% to 91% discount                |
+─────────────────────────────────────────────────────────────────────────────+

Why Secondary Workers Cannot Run HDFS DataNodes

Compute Engine Spot VMs can be reclaimed by Google at any moment with a 30-second preemption notice. If secondary workers ran HDFS DataNodes, a bulk preemption event (such as a sudden regional spike in Spot demand) would simultaneously terminate dozens of DataNodes. This would cause severe HDFS block loss, corrupted metadata, and pipeline failure. By running only YARN NodeManagers, secondary workers remain strictly stateless; when a Spot VM is reclaimed, YARN simply reassigns the affected task attempts to other surviving primary or secondary workers.

---

5. Cluster Initialization Actions

An Initialization Action is an executable shell script stored in Cloud Storage that Dataproc executes across cluster nodes during the provisioning workflow.

[ Provision Cluster ] ──> [ Execute Init Actions as root ] ──> [ Start Hadoop/YARN/Spark Daemons ] ──> [ Cluster RUNNING ]
                                      │
                                      ▼ If Script Exits with Non-Zero Code
                                [ Provisioning FAILS & Cluster Automatically Deleted ]

Operational Rules and Execution Semantics

• Execution Timing: Init actions run as root concurrently across all provisioned nodes after the operating system and networking boot, but before Hadoop, YARN, and Spark daemons are started.
• Failure Semantics: If any initialization action script fails (exits with a non-zero exit code), the Dataproc provisioning pipeline halts immediately, marks the cluster creation as failed, and automatically deletes all provisioned VMs. This fail-fast mechanism prevents users from paying for corrupted, half-configured clusters.
• Colocation Best Practice: Store initialization action scripts in a regional Cloud Storage bucket located in the exact same region as the Dataproc cluster to prevent cross-region latency, network egress fees, and transient network timeouts.
• Idempotency and Timeouts: Scripts should be deterministic and idempotent. Production deployments should configure --initialization-action-timeout (default: 10 minutes) to prevent hanging third-party package downloads from blocking cluster creation.

# Production Initialization Action Script: install-dependencies.sh
#!/bin/bash
set -euo pipefail

# Detect role of current node (Master vs Worker)
ROLE=$(/usr/share/google/get_metadata_value attributes/dataproc-role)

echo "Executing initialization action on node role: ${ROLE}"

# Install Python dependencies globally for PySpark
/opt/conda/default/bin/pip install --no-cache-dir \
    scikit-learn==1.3.0 \
    pandas==2.1.0 \
    google-cloud-bigquery==3.12.0

# Master-specific configurations
if [[ "${ROLE}" == 'Master' ]]; then
    echo "Configuring master-specific monitoring agents..."
    apt-get update && apt-get install -y jq
fi

---

6. Centralized Metadata: Dataproc Metastore (DPMS)

When operating ephemeral clusters, how do distributed engines retain table schemas, partitioned locations, and relational views? In traditional Hadoop, the Apache Hive Metastore (HMS) stores metadata in a local MySQL or PostgreSQL database running on the master node. When the cluster is terminated, the catalog is lost.

+─────────────────────────────────────────────────────────────────────────────+
|                 DATAPROC METASTORE (DPMS) CENTRAL CATALOG                   |
+─────────────────────────────────────────────────────────────────────────────+
|                                                                             |
|             [ Ephemeral Dataproc Cluster A (Morning ETL) ]                  |
|                                    │                                        |
|             [ Ephemeral Dataproc Cluster B (Ad-hoc Spark) ]                 |
|                                    │ (Connects via gRPC)                    |
|             [ BigQuery BigLake / Serverless Spark Engines ]                 |
|                                    │                                        |
|                                    ▼                                        |
|  +───────────────────────────────────────────────────────────────────────+  |
|  |                     DATAPROC METASTORE (DPMS)                         |  |
|  |  - Fully Managed, Serverless Apache Hive Metastore Service            |  |
|  |  - High Availability, Automated Backups, Point-in-Time Recovery       |  |
|  |  - Backed by Google-Managed Cloud SQL Database Instance               |  |
|  +───────────────────────────────────────────────────────────────────────+  |
|                                    │                                        |
|                                    ▼ Resolves Metadata For                  |
|  +───────────────────────────────────────────────────────────────────────+  |
|  |                     CENTRAL DATA LAKE STORAGE                         |  |
|  |                   gs://enterprise-data-lake/curated/                  |  |
|  +───────────────────────────────────────────────────────────────────────+  |
+─────────────────────────────────────────────────────────────────────────────+

Dataproc Metastore Service Architecture

• Managed HMS: Dataproc Metastore is a fully managed, serverless, highly available Hive metastore service based on open-source Apache Hive HMS.
• Protocol: Exposes standard Hive Metastore thrift/gRPC endpoints accessible over the customer's VPC network.
• Multi-Engine Sharing: Hundreds of independent ephemeral Dataproc clusters, persistent clusters, and BigQuery BigLake tables can query and register tables against the identical DPMS service simultaneously.
• Disaster Recovery: DPMS natively supports automated scheduled backups to Cloud Storage, metadata export/import, and point-in-time recovery without manual database maintenance.

---

7. Apache Spark Memory Model and Performance Tuning on Dataproc

A critical objective on the exam is diagnosing and fixing Spark performance bottlenecks and out-of-memory (OOM) failures.

+─────────────────────────────────────────────────────────────────────────────+
|                    APACHE SPARK EXECUTOR MEMORY LAYOUT                      |
+─────────────────────────────────────────────────────────────────────────────+
|  TOTAL YARN CONTAINER MEMORY ALLOCATION                                     |
|  [ spark.executor.memory ] + [ spark.executor.memoryOverhead ]              |
|                                                                             |
|  ┌──────────────────────────────────────────────┐ ┌──────────────────────┐  |
|  | spark.executor.memory (JVM Heap)             | | memoryOverhead       |  |
|  |                                              | | (Off-Heap / Native)  |  |
|  | ┌───────────────────────┐ ┌────────────────┐ | | - PySpark processes   |  |
|  | | Spark Memory (60%)    | | User Memory    | | | - Shuffle buffers    |  |
|  | | - Execution Memory    | | (40%)          | | | - Native C libraries |  |
|  | |   (Shuffles / Joins)  | | - User Data    | | | - JVM meta-space     |  |
|  | | - Storage Memory      | |   Structures   | | |                      |  |
|  | |   (Cached RDD/Tables) | | - Internal RDD | | | Default: Max(384MB,  |  |
|  | └───────────────────────┘ └────────────────┘ | |          10% of RAM) |  |
|  └──────────────────────────────────────────────┘ └──────────────────────┘  |
+─────────────────────────────────────────────────────────────────────────────+

Executor Sizing Rules of Thumb

When sizing Spark executors on Compute Engine worker VMs:

1. Do not assign all VM cores to a single executor: An executor with 16 or 32 cores suffers from severe JVM garbage collection (GC) pauses that halt pipeline processing.
2. Do not assign 1 core per executor: A 1-core executor destroys in-process multi-threading and eliminates the ability to share broadcast variables across tasks.
3. Optimal vCPU allocation: Allocate 4 to 5 vCPUs per executor (spark.executor.cores = 4 or 5) to balance I/O throughput and GC efficiency.
4. Memory Allocation: Divide total VM RAM by the number of executors per node, reserving at least 1 vCPU and 2 to 4 GB of RAM for OS and YARN NodeManager overhead.

Resolving "Container Killed by YARN for Exceeding Memory Limits"

This is the single most common Spark error on the Data Engineer exam:

• The Root Cause: In PySpark workloads, user code runs inside a separate Python worker process outside the JVM heap. Alternatively, operations using off-heap allocations, native C libraries, or heavy shuffle buffers consume memory in the container outside JVM heap space. When total container RAM exceeds spark.executor.memory + spark.executor.memoryOverhead, the YARN NodeManager terminates the container.
• The Solution: Increase spark.executor.memoryOverhead from its default (10% of executor memory or 384 MB) to 20%–30% (e.g., --properties spark.executor.memoryOverhead=2048m).

Partition Sizing and Shuffle Optimization

• Default Partitions Problem: spark.sql.shuffle.partitions defaults to 200. If you process 500 GB of data, each shuffle partition receives 2.5 GB of data, overwhelming executor RAM and causing severe disk spilling and OOM crashes. Conversely, processing a 50 MB dataset across 200 partitions creates microscopic partitions and excessive scheduling overhead.
• Optimal Partition Target: Target partition sizes of 100 MB to 200 MB uncompressed. Calculate optimal partitions using: $\text{spark.sql.shuffle.partitions} = \frac{\text{Total Input Data Size (MB)}}{128 \text{ MB}}$
• Adaptive Query Execution (AQE): In modern Dataproc images (Spark 3.0+), enable AQE (spark.sql.adaptive.enabled = true and spark.sql.adaptive.coalescePartitions.enabled = true) to dynamically coalesce small shuffle partitions at runtime.
• Broadcast Hash Joins: For joins between a large fact table and a small dimension table, configure spark.sql.autoBroadcastJoinThreshold (default: 10 MB, can be increased to 50 MB-100 MB). This broadcasts the small table to all executors, eliminating expensive network shuffles entirely.

---

8. Comparative Evaluation & Realistic Exam Scenarios