9.2 Deploying Pub/Sub, Dataflow & Loading Data into Google Cloud

Why This Section Matters

The exam guide's data-products bullet also names Pub/Sub and Dataflow among the things you must know how to deploy, and a separate bullet -- "Loading data (e.g., command line upload, load data from Cloud Storage, Storage Transfer Service)" -- tests how you move data into Google Cloud in the first place. Together these form the "plumbing" half of Domain 3's data content: messaging between services, transforming data in flight, and bulk-loading it from wherever it currently lives. Expect scenario questions that describe a data volume, source system, and freshness requirement, then ask you to pick the right ingestion tool -- confusing Storage Transfer Service (a managed copy job) with Dataflow (a transformation pipeline) is one of the most common ACE traps.

Pub/Sub -- Asynchronous Messaging

Pub/Sub decouples publishers from subscribers using two objects:

• A topic -- an append-only named resource that publishers send messages to.
• A subscription -- attached to a topic, delivers each message to its subscriber (a topic can have many subscriptions, giving fan-out to multiple independent consumers).

gcloud pubsub topics create order-events
gcloud pubsub subscriptions create order-events-sub \
  --topic=order-events --ack-deadline=30 \
  --message-retention-duration=7d

Delivery works in two modes:

Three deployment details the exam likes to test:

• Message retention: the default is 7 days after publish, and messages (acknowledged or not, depending on configuration) can be retained for up to 31 days, which lets a newly created subscription "replay" recent history instead of only seeing messages published after it existed.
• Ordering keys: attaching an ordering key to messages guarantees per-key delivery order, but push subscriptions allow only one outstanding message per ordering key at a time, which can add latency -- pull is usually preferred for high-volume ordered streams.
• Dead-letter topics: after a configurable maximum delivery attempts count, Pub/Sub forwards a message it cannot get acknowledged to a separate dead-letter topic for offline investigation, instead of retrying it forever.

Dataflow -- Managed Apache Beam Pipelines

Dataflow runs data-processing pipelines written against the open source Apache Beam model, in either batch mode (bounded data, such as a nightly file drop) or streaming mode (unbounded data, such as a live Pub/Sub topic) using the same programming model for both.

You rarely write a pipeline from scratch for common extract-transform-load (ETL) work: Google publishes ready-to-run Dataflow templates, and you launch one instead of authoring custom code:

gcloud dataflow jobs run pubsub-to-bq \
  --gcs-location=gs://dataflow-templates/latest/PubSub_Subscription_to_BigQuery \
  --region=us-central1 \
  --parameters inputSubscription=projects/my-proj/subscriptions/order-events-sub,outputTableSpec=my-proj:sales_ds.orders

There are two template families:

• Classic templates -- the original format; parameters are effectively fixed once the template is staged.
• Flex templates -- Google's current recommendation for any new template. The pipeline is packaged as a Docker image in Artifact Registry, referenced by a small specification file in Cloud Storage; this allows more flexible runtime parameters and support for arbitrary custom sources and sinks.

A common ACE-style scenario reads: "events land continuously in a Pub/Sub topic and must be transformed and written into BigQuery in near real time." The answer is a Dataflow streaming job -- ideally launched from a Google-provided template such as Pub/Sub Subscription to BigQuery -- not a manual bq load, which is a batch-only path that cannot consume a live topic. Note also that Google-managed template jobs now run on Dataflow Prime (an autotuning execution mode) by default; if a scenario calls for the older, standard Dataflow execution behavior, you disable Prime with the enable_prime=false service option.

Loading Data Into Google Cloud

The exam guide names three loading paths, each suited to a different volume and frequency profile:

A frequent exam trap is reaching for a custom Dataflow pipeline to "just copy files" from Amazon S3 into Cloud Storage -- that is exactly what Storage Transfer Service already does, managed and without custom code. Save Dataflow for when the data genuinely needs transformation in flight, not just relocation. For extremely large, one-time, offline migrations (measured in hundreds of terabytes or petabytes) where shipping data over the network is impractical, Google also offers the Transfer Appliance, a physical device you load on-site and ship back to Google -- rarely the right answer on the ACE exam, but useful to recognize as an outlier compared to the three primary loading paths above.