9.3 Event-Driven Ingestion: Pub/Sub to BigQuery & Eventarc Triggers

9.3 Event-Driven Ingestion: Pub/Sub to BigQuery & Eventarc Triggers

Core Focus: The exam guide asks you to identify use cases for event-driven ingestion from Pub/Sub to BigQuery and to use Eventarc triggers in event-driven pipelines. Event-driven designs start work when data arrives instead of on a clock, which cuts latency and avoids paying for jobs that find nothing to do.

Traditional pipelines run on fixed schedules such as a nightly cron job. That adds latency, wastes runs when no new data exists, and breaks when upstream producers are late. Event-driven pipelines react to a message or a file the moment it arrives.

---

Event-Driven vs. Time-Based Scheduling

Time-Based (Cron / Polling):
[ Clock: 02:00 UTC ] ---> (Triggers Pipeline) ---> [ Reads Cloud Storage Bucket ]
  - Problem 1: If data lands at 02:05 UTC, it waits almost 24 hours.
  - Problem 2: If data is late, the job processes empty or stale input.

Event-Driven (Pub/Sub / Eventarc):
[ File Lands in GCS ] ---> [ Eventarc: object finalized ] ---> (Triggers Pipeline)
  - Processing starts seconds after the upload completes.
  - Nothing runs (and nothing is billed) while waiting.

---

Pub/Sub Basics

Pub/Sub is Google Cloud's managed, asynchronous messaging service. Publishers send messages to a topic; each subscription on the topic receives its own copy of every message.

A dead-letter topic catches messages that fail delivery more than a configured number of attempts (5 by default, configurable from 5 to 100), so one bad message does not block the rest.

---

Three Ways to Get Pub/Sub Data into BigQuery

1. BigQuery Subscription (No Pipeline)

A BigQuery subscription writes each message to an existing BigQuery table as it arrives, using the Storage Write API behind the scenes. There is no Dataflow job or subscriber client to run.

• Schema options: use the topic schema (Avro or Protocol Buffers) mapped to table columns; use the table schema to map JSON message fields to matching columns; or, with neither option, write the raw message to a data column.
• Metadata: optionally write the subscription name, message ID, publish time, and attributes to extra columns for auditing.
• Failures: messages that cannot be written are negatively acknowledged and retried, and move to the dead-letter topic if one is configured.
• Delivery: at least once, so a downstream view or MERGE should deduplicate on message_id or a business key if duplicates matter.
• Light changes: a Single Message Transform (a small JavaScript function) can adjust a message before it is written.
• Permissions: the Pub/Sub service agent (or a service account you specify) needs permission to write to the table, such as BigQuery Data Editor on it.

gcloud pubsub subscriptions create orders-to-bq \
  --topic=orders \
  --bigquery-table=my-project:sales.orders_raw \
  --use-table-schema \
  --write-metadata \
  --dead-letter-topic=orders-dlq

2. Dataflow Streaming Pipeline

Use Dataflow (for example the Google-provided Pub/Sub to BigQuery template, or your own Apache Beam pipeline) when messages need real transformation: parsing, validation, enrichment with lookups, windowed aggregation, or routing bad records to a dead-letter output. The Pub/Sub-to-BigQuery template enforces exactly-once processing by default; a BigQuery subscription is at-least-once.

3. Custom Subscriber

A Cloud Run service on a push subscription can call the Storage Write API itself. Choose this only when you need custom logic that neither option above provides.

---

Eventarc: Routing Events to Pipelines

Eventarc routes events in the open CloudEvents format from Google Cloud services, Pub/Sub topics, and other sources to a destination.

• Direct events: For example, google.cloud.storage.object.v1.finalized, emitted when an upload to a bucket completes. Filters narrow it to one bucket.
• Cloud Audit Logs events: React to API calls recorded in audit logs (for example, a BigQuery job completing or a table being created) without changing the producer.
• Pub/Sub messages: Any message published to a topic can trigger a destination.
• Destinations: Cloud Run services, Cloud Run functions (formerly Cloud Functions), Workflows, GKE services, and internal HTTP endpoints in a VPC network.

Triggering Data Services from Eventarc

The exam guide mentions Eventarc with Dataform, Dataflow, Cloud Functions, Cloud Run, and Cloud Composer. Eventarc sends events to Cloud Run, Cloud Run functions, or Workflows, and that target then starts the data service through its API:

Loading diagram...

---

Failure Handling in Event-Driven Pipelines

1. Idempotency: Events can be delivered more than once. Use MERGE on a business key, deterministic job IDs, or deduplicating views so a retry does not double-count.
2. Dead-letter handling: Route messages that keep failing to a dead-letter topic or bucket instead of retrying forever, and alert on its size.
3. Backoff: Retry calls to rate-limited APIs (HTTP 429 or 503) with exponential backoff and jitter.

---

Common Exam Traps

• Building a Dataflow job just to copy messages into BigQuery: If no transformation is needed, a BigQuery subscription removes the pipeline entirely.
• Assuming exactly-once from a BigQuery subscription: It is at-least-once; deduplicate downstream when it matters.
• Polling a bucket: A function that lists a bucket every minute adds latency and API cost; an Eventarc trigger on object finalization starts work immediately.
• Silently dropping bad messages: Use a dead-letter topic so failures are kept, counted, and replayable.