6.1 Apache Spark and Hadoop with Cloud Dataproc

6.1 Apache Spark and Hadoop with Cloud Dataproc

[!IMPORTANT] A foundational principle tested on the Google Cloud Professional Data Engineer exam is the architectural decoupling of compute and storage. In legacy on-premises Hadoop environments, compute and storage were coupled on physical servers running HDFS. On Google Cloud, Dataproc separates these tiers by redirecting data storage to Google Cloud Storage (gs://). This separation allows clusters to be treated as ephemeral, disposable compute resources that can be spun up in 90 seconds, scaled elastically, and terminated the instant jobs finish.

Enterprise data architectures historically invested massive capital into on-premises Apache Hadoop and Apache Spark clusters. Maintaining those environments required dedicated operations teams to manage physical hardware failures, balance HDFS disk spindles, re-stripe blocks during node additions, and oversize compute capacity to satisfy occasional peak batch demands. Google Cloud Dataproc eliminates this administrative burden by delivering a fully managed, enterprise-ready service for open-source data processing engines—including Apache Spark, Hadoop YARN, Apache Hive, Apache HBase, Presto/Trino, and Apache Flink.

---

Decoupling Compute and Storage: The Cloud Storage Connector

The cornerstone of running Spark and Hadoop on Google Cloud is replacing the on-cluster Hadoop Distributed File System (HDFS) with Google Cloud Storage as the primary persistent data lake. Dataproc achieves this through the open-source Google Cloud Storage connector (gcs-connector), a Java library pre-installed in the Hadoop classpaths on every master and worker node.

+-------------------------------------------------------------------------+
|                        Cloud Dataproc Cluster                           |
|                                                                         |
|   +-------------------+    +--------------------+    +--------------+   |
|   |    Master Node    |    |  Primary Workers   |    |  Secondary   |   |
|   | (YARN RM, Driver) |    |  (YARN NodeMgr)    |    | Spot Workers |   |
|   +---------+---------+    +---------+----------+    +-------+------+   |
+-------------|------------------------|-----------------------|----------+
              |                        |                       |
              +------------------------+-----------------------+
                                       |
                            gcs-connector (gs://)
                                       |
                                       v
            +-----------------------------------------------------+
            |             Google Cloud Storage Bucket             |
            |   (11 9s Durability, Independent Scaling, Parquet)  |
            +-----------------------------------------------------+

Architectural Advantages of Cloud Storage over HDFS

1. Zero Data Loss on Cluster Termination: Because raw datasets, intermediate staging tables, and final transformed outputs live in Cloud Storage buckets (gs://bucket-name/path/), the compute cluster possesses no persistent state. Clusters can be stopped, deleted, or autoscaled down to zero secondary workers without risking permanent data loss.
2. Elimination of 3x Replication Disk Overhead: Standard HDFS enforces a minimum $3\times$ replication factor across local persistent disks to protect against drive and rack failures. Storing 500 TB of data in HDFS requires purchasing 1.5 PB of persistent disk capacity. In contrast, Cloud Storage natively provides $99.999999999%$ (11 nines) annual durability across redundant storage devices without charging for redundant copies.
3. Independent Elastic Scaling: In coupled architectures, expanding storage capacity requires adding more compute nodes, even if CPU utilization is near zero. With Dataproc, storage scales automatically to exabytes, while compute nodes scale strictly based on active YARN processing demand.
4. Concurrent Multi-Cluster Access: Multiple specialized Dataproc clusters can query the exact same Cloud Storage data lake simultaneously. For example, a persistent Trino cluster for interactive SQL queries, an ephemeral PySpark cluster for nightly machine learning feature extraction, and a Dataproc Serverless batch job can all read identical Parquet partitions in Cloud Storage without data movement or cross-cluster resource locking.

The Object Store Output Committer Problem and Solutions

Traditional Hadoop jobs write task outputs using the legacy FileOutputCommitter algorithm. In standard POSIX file systems or local HDFS, when a stage finishes, renaming temporary staging directories (_temporary/0/task_xyz/) into the final directory is an atomic metadata operation that executes in milliseconds.

However, Cloud Storage is an object store, not a hierarchical file system. There are no physical directories—only object keys containing slashes. In an object store, "renaming a directory" requires the client to:

1. Issue a LIST API call to discover every object matching the source prefix.
2. Execute a COPY API call for every individual file from the temporary path to the target path.
3. Execute a DELETE API call for every original temporary object.

For large-scale Spark jobs generating tens of thousands of partitioned files, this process creates an $O(N)$ performance bottleneck, frequently causing jobs to spend more time committing output files than actually crunching data. Furthermore, if a copy fails halfway through, the destination directory is left in an inconsistent, partially updated state.

Solutions for Exam Scenarios

• Dataproc Direct Output Committer / Manifest Committer: Bypasses temporary directory copies by writing output files directly to the final destination Cloud Storage path during task execution and generating an atomic metadata manifest upon overall stage success.
• Connector Buffer Tuning: For high-throughput sequential writes of Parquet, ORC, or Avro files, tune the connector configuration in core-site.xml or Spark submission properties:
  • fs.gs.block.size: Increase from the default 64 MB to 128 MB (134217728) or 256 MB to optimize large columnar block scans.
  • fs.gs.io.buffersize.write: Increase write buffer memory (e.g., 64 MB) to reduce HTTP POST chunking overhead against Cloud Storage APIs.

---

Cluster Provisioning Models: Persistent vs. Ephemeral vs. Dataproc Serverless

A central decision on the exam is selecting the appropriate execution and lifecycle model for a given data engineering workload.

+---------------------------------------------------------------------------------+
|                        Dataproc Provisioning Spectrum                           |
+-----------------------+---------------------------------+-----------------------+
|  Persistent Clusters  |       Ephemeral Clusters        |  Dataproc Serverless  |
+-----------------------+---------------------------------+-----------------------+
| • 24/7 Uptime         | • Created on demand per job     | • Zero VM management  |
| • Shared multi-tenant | • Sized for exact workload      | • Per-second billing  |
| • High idle cost      | • Deleted immediately upon done | • Auto-tuned Spark    |
| • Interactive BI/SQL  | • Scheduled batch ETL / ELT     | • Event-driven batch  |
+-----------------------+---------------------------------+-----------------------+

1. Persistent (Long-Running) Clusters

Persistent clusters run continuously across weeks or months. They maintain master and primary worker VMs 24/7.

• When to Use: Interactive data exploration environments where dozens of data scientists share JupyterLab or Zeppelin notebooks; low-latency enterprise BI environments hosting distributed SQL engines like Trino/Presto or Hive LLAP; and continuous streaming jobs (e.g., Spark Structured Streaming reading from Kafka or Pub/Sub).
• Exam Pitfalls: High idle infrastructure cost when no jobs are running; configuration drift and disk clutter over time; and YARN queue contention ("noisy neighbors") where a runaway query starves other users of cluster memory.

2. Ephemeral (Job-Scoped) Clusters

Ephemeral clusters are instantiated dynamically for the specific lifecycle of a single workflow or batch job and torn down immediately after execution.

• Lifecycle Flow: An orchestrator (such as Cloud Composer or Cloud Workflows) calls the Dataproc API to create a cluster $\rightarrow$ Dataproc provisions nodes in ~90 seconds $\rightarrow$ The orchestrator submits the Spark, PySpark, or Hive job $\rightarrow$ The job completes and writes results to Cloud Storage $\rightarrow$ The orchestrator executes an API call to delete the cluster.
• When to Use: Scheduled batch ETL/ELT pipelines; large nightly data consolidation jobs; and specialized machine learning workloads requiring unique GPU hardware configurations.
• Benefits: Complete elimination of idle compute costs; perfect job isolation (each job has its own dedicated YARN ResourceManager and network bandwidth); and right-sized hardware tailored to the specific pipeline step.

3. Dataproc Serverless for Spark

Dataproc Serverless represents Google Cloud's modern, zero-infrastructure computing model for Apache Spark. Instead of provisioning Compute Engine clusters, engineers submit Spark code directly:

gcloud dataproc batches submit spark \
    --batch=daily-sales-transform-20260914 \
    --class=com.enterprise.analytics.SalesAggregator \
    --jars=gs://analytics-binaries/sales-pipeline.jar \
    --version=2.2 \
    --region=us-central1 \
    --subnet=analytics-vpc-subnet \
    --deps-bucket=gs://analytics-staging-bucket

• Mechanics: Google Cloud dynamically manages the Spark driver, allocates executors, handles dynamic auto-scaling, and manages container isolation. Billing is calculated per second based on Data Compute Units (DCUs) consumed by the driver and executors.
• Custom Container Images: If dependencies are required, engineers specify a custom container image hosted in Artifact Registry (--container-image=...), bypassing VM boot overhead.
• When to Use: Event-driven Spark pipelines triggered by Cloud Storage file arrivals (via Eventarc and Cloud Functions); ad-hoc PySpark scripts; and teams that want pure Spark processing without managing Compute Engine infrastructure, OS patching, or YARN configurations.

Architectural Comparison Matrix

---

Cluster Topology and High Availability (HA) Architecture

Dataproc supports three primary cluster deployment modes:

1. Single Node (Developer Mode): Master and worker daemons run on a single Compute Engine VM (--single-node). HDFS, YARN ResourceManager, and NodeManager share memory. Strictly for developer prototyping, unit testing, and small CI/CD validations. Never use in production.
2. Standard Mode (1 Master, N Workers): A single master node hosts the YARN ResourceManager, HDFS NameNode, and job history daemons. If the master VM fails, the cluster is disrupted until Compute Engine restarts the instance. Suitable for ephemeral batch jobs where pipeline failures can be cleanly retried by an orchestrator.
3. High Availability Mode (3 Masters, N Workers): Deploys three master nodes running Apache ZooKeeper and Quorum Journal Manager (QJM) across separate Compute Engine instances (--num-masters=3). One master acts as the Active YARN ResourceManager and Active HDFS NameNode, while the other two remain in Standby state, continuously syncing transaction logs.
   • Automatic Failover: If the active master VM experiences hardware failure or zone disruption, ZooKeeper elects a standby master to become active within seconds, allowing in-flight jobs to continue without failure.
   • Production Requirement: Mandatory for long-running, mission-critical persistent clusters hosting shared BI engines or continuous streaming applications.

Component Gateway and Web UI Access

Historically, accessing web interfaces like the YARN ResourceManager (port 8088), Spark History Server (port 18080), JupyterLab, or Apache Zeppelin required configuring SSH tunnels, SOCKS proxies in web browsers, or opening vulnerable firewall ports to the public internet.

Dataproc's Component Gateway resolves this security hazard:

• Enabled during cluster creation via --enable-component-gateway.
• Provides direct, secure web URL endpoints for cluster UIs hosted via Google Cloud's Knox reverse proxy.
• Authenticates requests natively via Google Cloud IAM permissions (roles/dataproc.editor or roles/dataproc.viewer), eliminating the need for SSH tunneling or external IP addresses.

---

Cluster Customization: Initialization Actions vs. Custom Images

Production clusters often require specialized Python libraries (e.g., PyTorch, scikit-learn, proprietary internal SDKs), C/C++ native binaries, monitoring agents, or database drivers.

Initialization Actions

Initialization actions are executable bash scripts stored in Cloud Storage that Dataproc executes on every node (master and workers) during cluster provisioning before Hadoop and Spark daemons start:

gcloud dataproc clusters create analytics-cluster \
    --region=us-central1 \
    --master-machine-type=n2-standard-4 \
    --worker-machine-type=n2-standard-8 \
    --num-workers=4 \
    --initialization-actions=gs://my-bucket/scripts/install-deps.sh \
    --initialization-action-timeout=15m

• Operational Mechanics: Scripts run as root. Engineers can pass cluster metadata variables to dynamically alter behavior between master and worker nodes (/usr/share/google/get_metadata_value attributes/dataproc-role).
• Best Practices: Scripts must be strictly idempotent (safe to execute multiple times). Buckets storing initialization scripts must be co-located in the same Google Cloud region as the cluster to prevent cross-region latency.
• The Autoscaling Failure Trap: If an initialization action downloads packages from public internet repositories (e.g., pip install ... or apt-get install ...), it introduces external dependencies. If a PyPI mirror experiences a transient outage or network throttling during an autoscaling scale-up event, newly spawned worker nodes will fail to initialize. The cluster autoscaler will abort, leaving the cluster starved of capacity during peak workloads. Furthermore, compiling C++ wheels or large packages can extend node boot times to 15–20 minutes, neutralizing the agility of autoscaling.

Custom Dataproc Images

To ensure deterministic, rapid, and enterprise-grade cluster provisioning, Google Cloud recommends building Custom Dataproc Images using the open-source dataproc-custom-images tool:

• Baking Dependencies: Pre-installs and pre-compiles all system packages, Python libraries, kernels, and security monitoring daemons directly into a customized Compute Engine boot disk image.
• Launch Latency: Reduces node creation and autoscaling boot times to approximately 90 seconds, regardless of the size or complexity of dependencies.
• Air-Gapped & VPC Service Controls Compliance: Essential for secure enterprise perimeters without public internet access. Because all dependencies are baked into the image, nodes initialize cleanly without attempting outbound connections to external package repositories.

---

Cost Optimization: Secondary Workers, Spot VMs, and Graceful Decommissioning

Compute Engine costs account for the vast majority of big data operational spend. Dataproc divides cluster nodes into three distinct functional tiers to unlock aggressive cost reductions:

+---------------------------------------------------------------------------------+
|                           Dataproc Worker Role Matrix                           |
+------------------------------------+--------------------------------------------+
|     Primary Workers (On-Demand)    |        Secondary Workers (Spot VMs)        |
+------------------------------------+--------------------------------------------+
| • Runs YARN NodeManager            | • Runs YARN NodeManager                    |
| • Runs HDFS DataNode daemon        | • NO HDFS DataNode (Pure Compute)          |
| • Hosts YARN ApplicationMaster     | • Transient Spark Executor tasks only      |
| • Standard on-demand pricing       | • 60% to 91% cost discount via Spot VMs    |
| • Cannot be preempted              | • Subject to Compute Engine reclamation    |
| • Maintains core cluster stability | • Graceful decommissioning prevents data   |
|                                    |   loss during preemption                   |
+------------------------------------+--------------------------------------------+

Spot (Preemptible) VMs for Secondary Workers

In Dataproc, secondary workers are stateless, compute-only nodes. They execute YARN container tasks and Spark executors, but they never run HDFS DataNodes and never host YARN Application Masters.

• Deploying Spot VMs: By configuring --secondary-worker-type=SPOT, secondary workers utilize Google Cloud Spot VMs, delivering 60% to 91% savings compared to standard VM pricing.
• Fault Tolerance: If Google Cloud reclaims a Spot VM, the active Spark job does not crash. The master node detects the loss of the YARN NodeManager and automatically reschedules interrupted task attempts on surviving workers.

The Intermediate Shuffle Loss Problem and Graceful Decommissioning

While Spark can recompute failed tasks, losing a worker that holds intermediate shuffle partitions introduces severe performance penalties. In Apache Spark, wide transformations (such as groupBy, join, and reduceByKey) require data to be partitioned and written to the worker's local scratch disks before downstream stages fetch it over the network.

If a secondary Spot VM is abruptly terminated while hosting intermediate shuffle files:

1. Downstream reducer tasks attempting to read those shuffle partitions encounter fetch-failure exceptions.
2. Spark is forced to roll back execution and recompute the entire upstream stage that produced those partitions from scratch.
3. In heavy iterative workloads (such as graph analytics or iterative machine learning algorithms), repeated Spot preemptions can trap a pipeline in an infinite loop of stage recalculations, causing the job to fail.

Graceful Decommissioning Mechanics

Dataproc mitigates shuffle loss through Graceful Decommissioning (--graceful-decommission-timeout):

• When Google Cloud issues a preemption notice or the autoscaler decides to scale down, Dataproc instructs YARN to stop assigning new container tasks to the targeted worker.
• The worker is granted a configurable grace period (e.g., 10 to 60 minutes) to conclude currently running tasks and replicate its intermediate shuffle partitions and cached RDD/DataFrame blocks to surviving primary workers or Cloud Storage.
• If all tasks finish and shuffle data is safely evacuated before the timeout expires, the node shuts down cleanly without triggering stage recalculations.

---

Autoscaling Policies and YARN Metric Tuning

Dataproc provides native horizontal autoscaling to adjust cluster size dynamically during variable batch workloads. Unlike standard Compute Engine instance group autoscalers that evaluate average CPU utilization, the Dataproc autoscaler evaluates Hadoop YARN resource demand metrics:

• yarn:allocated_memory_percentage: The ratio of cluster memory currently allocated to running YARN containers.
• yarn:pending_memory: The aggregate volume of memory requested by submitted YARN applications that cannot be scheduled due to insufficient resources.
• yarn:containers: The count of active, allocated, and pending containers across the cluster.

Autoscaling Policy YAML Configuration

Autoscaling behavior is defined via an autoscaling policy resource:

basicAlgorithm:
  yarnConfig:
    scaleUpFactor: 0.1
    scaleDownFactor: 0.05
    scaleUpMinWorkerFraction: 0.0
    scaleDownMinWorkerFraction: 0.0
    gracefulDecommissionTimeout: 1800s
  cooldownPeriod: 120s
workerConfig:
  minInstances: 2
  maxInstances: 4
secondaryWorkerConfig:
  minInstances: 0
  maxInstances: 60
  weight: 1.0

Key Parameters and Tuning Guidelines for the Exam

1. scaleUpFactor and scaleDownFactor: Controls how aggressively the autoscaler adds or removes instances based on pending YARN memory. A scaleUpFactor of 0.1 adds 10% of the pending instance requirement per evaluation cycle.
2. cooldownPeriod: The minimum waiting time (e.g., 120 seconds) between successive scaling actions. Prevents "thrashing" (rapid cycles of adding and removing nodes due to momentary metric spikes).
3. Primary vs. Secondary Instance Allocation: In production cost-optimized pipelines, set workerConfig (primary workers) to a small, fixed baseline (e.g., minInstances: 2, maxInstances: 2 or 4) running standard on-demand VMs. Set secondaryWorkerConfig with minInstances: 0 and maxInstances: 60 using Spot VMs with weight: 1.0. This architecture guarantees that 100% of dynamic scaling capacity utilizes discounted Spot compute.
4. gracefulDecommissionTimeout: Must be configured with a realistic timeframe (e.g., 30m or 1h) to allow active Spark shuffle operations to evacuate safely during scale-down.

---

Spark Performance Tuning on Dataproc

Achieving maximum throughput and preventing out-of-memory errors on Dataproc requires tuning Spark hardware and memory properties:

Memory Allocation Hierarchy

When Spark runs on YARN, each worker node's physical RAM is partitioned:

• OS and Daemon Overhead: Approximately 10% to 15% of node RAM is reserved for the Linux OS, YARN NodeManager, and Dataproc agents.
• YARN Container Memory: The remaining RAM is managed by YARN (yarn.nodemanager.resource.memory-mb).
• Executor Memory (spark.executor.memory): The JVM heap memory allocated to individual Spark executor tasks.
• Memory Overhead (spark.yarn.executor.memoryOverhead): Off-heap memory allocated for JVM native allocations, internal string operations, and PySpark Python worker processes. By default, it is $\max(384\text{ MB}, 0.10 \times \text{executorMemory})$. In PySpark pipelines utilizing heavy C libraries (NumPy, Pandas), this value must be explicitly increased to prevent YARN from killing executor containers for exceeding memory limits.

Storage Disks for Shuffle I/O

When Spark executes wide transformations, intermediate shuffle partitions are spilled to local worker disks. Using standard Persistent Disks (PD-standard) introduces severe I/O bottlenecks. For shuffle-heavy production workloads:

• Attach Local SSDs (--num-worker-local-ssds=1 or 2) to worker nodes to provide high-IOPS, ultra-low-latency NVMe scratch space for Spark shuffle spillover.
• Alternatively, use Balanced Persistent Disks (pd-balanced) for a balance of cost and performance.