4.1 Cloud Pub/Sub Core Architecture and Ingestion

4.1 Cloud Pub/Sub Core Architecture and Ingestion

[!IMPORTANT] On the Google Cloud Professional Data Engineer exam, Cloud Pub/Sub serves as the primary ingestion backbone for real-time streaming architectures. Key exam scenarios test your ability to select the correct subscription type (StreamingPull vs. Push vs. Direct Export), enforce secure webhook authentication via OpenID Connect (OIDC), configure message retention for zero-data-loss replay, and achieve deterministic message ordering without crippling ingestion throughput.

Modern enterprise data platforms must ingest high-velocity, heterogeneous event streams from distributed sources—including Internet of Things (IoT) sensor fleets, mobile application clickstreams, e-commerce checkout events, transactional database mutations, and microservice audit logs. These ingestion architectures must absorb massive traffic spikes without dropping records, introduce zero tight coupling between producers and consumers, and scale elastically across global geographical footprints.

Google Cloud Pub/Sub addresses these requirements through a fully managed, serverless publish/subscribe messaging infrastructure designed to process hundreds of millions of messages per second with single-digit millisecond latency. Understanding its internal topology, subscription mechanisms, delivery semantics, and operational controls is essential for designing resilient data pipelines on Google Cloud.

---

Distributed Messaging Architecture: Topics, Subscriptions, and Messages

Cloud Pub/Sub decouples systems that produce events from systems that process events. Unlike traditional on-premises message brokers (such as RabbitMQ or active/passive JMS queues) that bind queues directly to worker processes and require manual cluster sizing, Pub/Sub separates ingestion endpoints from message consumption channels through three core primitives:

+--------------------+         +--------------------+         +------------------------+
|   Event Producer   | ------> |  Pub/Sub Topic     | ------> | Subscription A (Pull)  | ---> Dataflow Pipeline
|  (IoT / App Log)   | Publish | (Global Ingestion) | Routing +------------------------+
+--------------------+         +--------------------+    |    +------------------------+
                                                         +--> | Subscription B (Push)  | ---> Cloud Run Webhook
                                                              +------------------------+
                                                              +------------------------+
                                                              | Subscription C (Direct)| ---> BigQuery Table
                                                              +------------------------+

1. Topics

A Topic is a named, globally accessible resource to which publishers send messages. Topics do not store messages independently; instead, a topic acts as a logical ingestion gateway that instantly routes published messages across all attached subscriptions. Topics automatically distribute traffic across Google Cloud's global network of forwarder and router servers, scaling horizontally without manual partition provisioning, broker rebalancing, or cluster capacity management.

When a message is published, the Pub/Sub forwarder layer persists the payload redundantly across multiple availability zones within the target region (or multi-region) using Google's distributed Colossus storage filesystem before acknowledging receipt to the publisher. This guarantees immediate multi-zone durability and prevents data loss in the event of an infrastructure zone outage.

2. Subscriptions

A Subscription represents an independent, stateful stream of messages originating from a single topic. Multiple subscriptions can attach to the same topic, implementing the fan-out pattern where each subscriber group receives its own complete, isolated copy of the message stream. For example, a single orders-v1 topic can fan out simultaneously to a real-time fraud detection Dataflow pipeline, a Cloud Storage archival export subscription, and a Cloud Run notifications webhook.

Conversely, multiple worker instances reading from the same subscription implement the competing consumers pattern, automatically load-balancing message processing across all active workers. Pub/Sub dynamically assigns message leases among active worker threads, ensuring that no two workers concurrently process the same message under normal operation.

Server-Side Message Filtering

Subscriptions support server-side message filtering. By specifying a filter expression using Pub/Sub attribute syntax, the Pub/Sub routing plane evaluates and drops non-matching messages before dispatch:

attributes.event_type = "PAYMENT_SETTLED" AND attributes.currency = "USD"

Key architectural characteristics of server-side filtering on the exam include:

• Bandwidth and Compute Optimization: Filtering occurs entirely within the Pub/Sub service plane. Non-matching messages are automatically acknowledged by the system without being transmitted over the network to the consumer, saving egress bandwidth and worker CPU cycles.
• Filterable Fields: Filters operate on message attributes (e.g., attributes.key = "value", hasPrefix(attributes.key, "prefix"), or boolean checks) and certain system attributes (such as attributes.type). Message payloads cannot be inspected for filtering.
• Billing Impact: Filtered-out messages are billed as standard subscription delivery throughput. However, downstream egress fees and consumer compute costs are completely avoided.

3. Messages

A Pub/Sub Message contains the following components:

• Payload (data): The core message body, up to 10 MB in size, represented as a base64-encoded byte array. Payloads commonly contain serialized JSON, Apache Avro, or Protocol Buffer records.
• Attributes (attributes): An optional key-value dictionary of string pairs (up to 100 attributes per message, maximum 1,024 bytes per key/value pair). Attributes convey routing metadata, tenant identifiers, schema versions, or filtering criteria.
• Message ID (messageId): A globally unique string assigned by the Pub/Sub service upon successful publish confirmation.
• Publish Timestamp (publishTime): A server-assigned UTC timestamp marking the exact millisecond when the message was persisted and acknowledged to the publisher.
• Ordering Key (orderingKey): An optional string identifier used to enforce FIFO delivery among messages sharing the same key.

---

Subscription Types and Delivery Models

Choosing between Pull, Push, and Direct Export delivery models is a central architectural decision evaluated on the certification exam. The decision depends entirely on consumer architecture, throughput requirements, network topology, and processing latency SLAs.

                    +-----------------------------------------------------+
                    |             Cloud Pub/Sub Subscription              |
                    +-----------------------------------------------------+
                         |                     |                     |
           [StreamingPull / Sync Pull]   [HTTPS POST Push]    [Direct Export Push]
                         |                     |                     |
                         v                     v                     v
            +---------------------+   +-----------------+   +-----------------+
            |  Consumer Workers   |   | HTTPS Endpoint  |   | Managed Sink    |
            | (Dataflow / GKE)    |   | (Cloud Run / App|   | (BigQuery / GCS)|
            |   Pulls Messages    |   | Receives Push   |   | Direct Storage  |
            +---------------------+   +-----------------+   +-----------------+

1. Pull Subscriptions

In a pull subscription, the subscribing client initiates all communication with the Pub/Sub service by requesting messages over Remote Procedure Calls (RPC):

A. StreamingPull (Standard for Real-Time Stream Ingestion)

• Mechanism: Established via a persistent, bidirectional gRPC stream (StreamingPullRequest and StreamingPullResponse) operating over HTTP/2 multiplexed TCP connections.
• Throughput & Latency: The client opens persistent channels over which the Pub/Sub server pushes batches of messages the instant they are published, yielding sub-second delivery latency with zero polling overhead.
• Automated Lease Management: Modern Google Cloud client libraries automatically maintain background heartbeat threads. While a worker thread is executing, the client library automatically issues modifyAckDeadline requests in the background, extending the message's acknowledgement deadline dynamically until processing finishes.
• Recommended Use Cases: High-throughput streaming workloads requiring hundreds of megabytes per second, including Cloud Dataflow (Apache Beam), Apache Spark on Cloud Dataproc, and large-scale Google Kubernetes Engine (GKE) microservice fleets.

B. Synchronous Pull (Unary Pull)

• Mechanism: Relies on standard unary RPC requests (PullRequest), where the client requests up to maxMessages and blocks until messages arrive or a timeout occurs.
• Throughput & Latency: High polling latency and connection establishment overhead. If the subscription backlog is empty, workers spend compute cycles issuing empty pull requests.
• Recommended Use Cases: Periodic batch jobs, Cloud Run scheduled jobs, lightweight administrative scripts, or legacy applications that process discrete micro-batches on cron schedules.

2. Push Subscriptions

In a push subscription, Cloud Pub/Sub acts as an HTTP client, delivering messages by sending HTTPS POST requests containing JSON-wrapped message envelopes to an HTTPS webhook endpoint:

{
  "message": {
    "attributes": {
      "event_type": "ORDER_PLACED"
    },
    "data": "eyJvcmRlcl9pZCI6ICJPUkQtODkwMSIsICJhbW91bnQiOiAxNDkuOTV9",
    "messageId": "1204981290381029",
    "publishTime": "2026-09-14T17:00:00.123456Z"
  },
  "subscription": "projects/my-proj/subscriptions/order-webhook-sub"
}

Endpoint Authentication via OpenID Connect (OIDC)

To protect webhook endpoints from unauthorized public invocations, push subscriptions integrate natively with Google Cloud IAM using OpenID Connect (OIDC):

1. Configuration: The push subscription is configured with a dedicated user-managed service account (serviceAccountEmail) and an optional audience string (audience).
2. Token Generation: Pub/Sub generates a Google-signed JWT bearer token placed in the HTTP Authorization: Bearer <JWT_TOKEN> header of every outbound POST request.
3. Validation at Ingress: The receiving service (such as Cloud Run or Cloud Functions) verifies the signature against Google's public keys. Cloud Run automatically enforces this when ingress authentication is enabled, checking that the invoking service account possesses the roles/run.invoker permission.

Acknowledgement Semantics and HTTP Status Codes

• Success (200, 201, 202, 204): Pub/Sub considers the message delivered and marks it acknowledged (ACK).
• Transient Failure (5xx, network timeout, or retryable 4xx): Pub/Sub considers delivery failed (NACK) and reschedules the message for redelivery according to the subscription retry policy.
• Permanent Dropped (404 Not Found, 410 Gone): Pub/Sub marks the message acknowledged and drops it permanently, preventing infinite retries against deleted endpoints.

3. Direct Export Subscriptions (BigQuery and Cloud Storage)

Google Cloud provides specialized direct-export push subscriptions that stream messages directly into managed analytical sinks without requiring intermediate compute engines (such as Cloud Functions, Cloud Run, or Dataflow):

A. BigQuery Export Subscriptions

• Functionality: Writes message payloads directly into a target BigQuery table using the BigQuery Storage Write API.
• Configuration Options: Supports writing message metadata fields (publish_time, message_id, attributes, subscription_name) into reserved table columns. Can write the raw payload to a data column or map JSON fields directly to BigQuery table columns using topic schemas.
• IAM Requirements: The Google-managed Pub/Sub Service Agent (service-<PROJECT_NUMBER>@gcp-sa-pubsub.iam.gserviceaccount.com) must be granted roles/bigquery.dataEditor on the destination table and roles/bigquery.metadataViewer on the dataset.
• Cost Advantage: Incurs only Pub/Sub egress and BigQuery storage fees; zero intermediate VM or container compute costs are incurred.

B. Cloud Storage Export Subscriptions

• Functionality: Batches messages and writes them directly into Cloud Storage buckets as files in Apache Avro or Text/JSON format.
• Batching Parameters: Configurable by maximum time duration (e.g., flush every 5 minutes) and maximum file size (e.g., flush at 100 MB).
• Path Formatting: Dynamic path variable prefixes allow structuring objects by date, hour, and minute (e.g., gs://my-bucket/logs/year=%Y/month=%m/day=%d/).
• IAM Requirements: The Pub/Sub Service Agent requires roles/storage.objectCreator on the target bucket.

---

Message Lifecycle, Retention Policies, and Replay (Seek)

Understanding how Pub/Sub tracks message lifecycle states prevents data loss and empowers automated pipeline disaster recovery.

+-------------+
|  Published  | ---> Persisted redundantly across availability zones in Colossus
+-------------+
       |
       v
+-------------+
| Dispatched  | ---> Delivered to subscriber; Ack Deadline timer begins
+-------------+
       |
       +-------------------------------------+
       |                                     |
       v (Worker completes task)             v (Timeout / Worker crashes)
+-------------+                       +-------------+
|     ACK     |                       | NACK / Exp  |
+-------------+                       +-------------+
       |                                     |
       | (retainAckedMessages = true)         +---> Redelivered with backoff
       v
+-----------------------------------+
| Retained up to 31 Days for Replay |
| (Seek to Timestamp or Snapshot)   |
+-----------------------------------+

The Acknowledgement Deadline and Lease Management

When Pub/Sub delivers a message to a subscriber, it starts an acknowledgement deadline timer (configurable per subscription between 10 and 600 seconds, defaulting to 10 seconds). The subscriber must process the message and return an ACK before the deadline expires. If the subscriber crashes, hangs, or explicitly returns a negative acknowledgement (NACK), Pub/Sub releases the message lock and redelivers it.

Modern Google Cloud client libraries implement automatic lease management:

• While subscriber worker threads remain actively executing a task, the client library issues background modifyAckDeadline requests to extend the deadline dynamically up to max_extension_duration (configurable up to several hours).
• If a worker crashes or encounters an unhandled fatal error, the heartbeat halts immediately. The deadline lapses, and Pub/Sub re-dispatches the message to a surviving worker node.

Retention Policies

1. Unacknowledged Message Retention: Unacknowledged messages are retained in the subscription backlog for a configurable period between 10 minutes and 31 days (defaulting to 7 days). If no subscriber consumes them before the retention period expires, messages are permanently purged from the system.
2. Acknowledged Message Retention (retainAckedMessages = true): By default, Pub/Sub deletes a message immediately upon receiving an ACK. When retainAckedMessages is enabled on a subscription, Pub/Sub preserves acknowledged messages for the entire subscription retention window (up to 31 days). This retained buffer enables the Seek feature.

The Seek Feature: Replaying Messages and Snapshots

The Seek capability allows administrators and automated deployment scripts to rewind a subscription's delivery cursor:

• Seek to Timestamp: Rewinds the subscription backlog to an exact historical UTC timestamp (e.g., "rewind cursor to 3 hours ago"). Any message published after that timestamp—whether previously acknowledged or unacknowledged—is marked unacknowledged and re-dispatched to active subscribers.
• Seek to Snapshot: A Snapshot captures the exact unacknowledged state of a subscription at a specific instant. If an engineering team deploys a new version of a streaming Dataflow pipeline and discovers a transformation calculation bug 4 hours later, the team can deploy a code fix and execute a seek back to the pre-deployment snapshot. This cleanly reprocesses the exact event stream without dropping records or introducing data gaps.
• Backlog Purging: In an incident where a faulty upstream producer floods a topic with millions of invalid test messages, an administrator can execute a seek to the current timestamp (now), immediately acknowledging and purging all backed-up messages to unblock consumers.

---

Exactly-Once Delivery Semantics vs. At-Least-Once Default

By default, Cloud Pub/Sub guarantees at-least-once delivery. In distributed systems, temporary network splits, subscriber crashes, or acknowledgement timeouts can cause a message to be successfully processed by a worker while the corresponding ACK is lost in transit. Pub/Sub's safety guarantee ensures no message is ever lost, which means duplicate deliveries can occur.

Cloud Pub/Sub Exactly-Once Delivery

For pull subscriptions, Google Cloud offers a managed Exactly-Once Delivery feature. When enabled on a subscription:

1. Ack Deduplication: Pub/Sub ensures that once a message is acknowledged, any subsequent redelivery attempts are suppressed across the entire global routing layer.
2. In-Flight Deduplication: If a worker's acknowledgement deadline is being actively extended via StreamingPull lease management, Pub/Sub guarantees it will not deliver that same message to any other concurrent worker instance.
3. Success / Failure Ack Status: When a subscriber invokes acknowledge(), the gRPC response confirms whether the acknowledgment was accepted or whether the deadline had already elapsed, preventing silent double-writes.

[!NOTE] While Pub/Sub exactly-once delivery eliminates duplicate deliveries at the subscription boundary (within a 10-minute deduplication window), downstream sinks that require absolute transactional idempotency (such as inserting into BigQuery or relational databases) should still implement idempotent write patterns (e.g., using record primary keys, Spanner upserts, or BigQuery deterministic insert IDs).

---

Message Ordering and Ordering Keys

In standard topics, Pub/Sub maximizes global ingestion throughput by spreading messages across thousands of internal message broker shards. Consequently, messages are delivered to subscribers in arbitrary order, regardless of publish sequence.

However, transactional workloads—such as financial ledger debits and credits, inventory mutations, or user profile updates—demand strict sequential execution.

Publisher sends events with Ordering Key = "acct_4502":
  [Event 1: Create Account] ---> [Event 2: Deposit $500] ---> [Event 3: Withdraw $200]
                                      |
                                      v
                         Pub/Sub Ordering Guarantee
                                      |
                                      v
  Subscriber receives: Event 1  ==>  Subscriber receives: Event 2  ==>  Subscriber receives: Event 3
  (Event 2 is NEVER delivered until Event 1 is successfully ACKed)

How Ordering Keys Function

1. Publishing with Keys: The publisher sets the orderingKey string attribute (e.g., user_id, device_serial_number, or account_id) on each published message.
2. Subscription Ordering Enabled: The subscription must be configured with enableMessageOrdering = true.
3. Guaranteed Sequential Delivery: Pub/Sub ensures that messages bearing the same orderingKey are delivered to subscribers strictly in the order they were received and timestamped by the Pub/Sub service.
4. Per-Key Scalability: Ordering is enforced strictly per key, not across the entire topic. Two messages with different ordering keys (e.g., acct_4502 and acct_8911) are delivered concurrently in parallel across multiple worker threads. This provides linear horizontal scaling across millions of distinct entity IDs.
5. Head-of-Line Blocking Failure Semantics: If an error occurs while processing a message for an ordering key, or if the message is explicitly NACK'd, Pub/Sub halts delivery of all subsequent messages for that specific ordering key. Delivery remains paused until the blocking message is successfully processed and acknowledged, or until the message is routed to a dead-letter topic or purged via Seek. This behavior prevents state corruption and out-of-order execution in downstream databases.

---

Exam Traps and Antipatterns Summary