5.1 Cloud Pub/Sub Core Architecture: Topics, Subscriptions, and Message Lifecycle

5.1 Cloud Pub/Sub Core Architecture: Topics, Subscriptions, and Message Lifecycle

Exam Focus: The Google Cloud Professional Data Engineer exam rigorously tests your understanding of Pub/Sub as an asynchronous buffering layer in event-driven and streaming analytics pipelines. You must know how to choose between Pull and Push delivery models, tune acknowledgement deadlines (ackDeadlineSeconds) to prevent duplicate processing, configure secure OpenID Connect (OIDC) authentication for serverless push endpoints, manage message retention boundaries, and execute Seek operations using timestamps and snapshots for disaster recovery and stream replay.

At the core of modern event-driven data platforms on Google Cloud lies Cloud Pub/Sub, an enterprise-grade, fully managed real-time messaging service. Pub/Sub acts as a resilient buffer between upstream producers (IoT edge devices, transaction logs, application clickstreams) and downstream consumers (Cloud Dataflow, BigQuery, Cloud Storage, Cloud Functions, and Kubernetes workloads). Understanding its internal architecture, decoupled storage mechanisms, and lifecycle states is essential for building fault-tolerant, high-throughput data architectures.

---

1. Distributed Pub/Sub Architecture: Routers, Forwarders, and Decoupled Storage

Unlike traditional messaging brokers that couple connection handling, message queueing, and local disk storage on individual virtual machine brokers, Cloud Pub/Sub is architected as a horizontally scalable, decoupled distributed system comprising two distinct planes:

1. Stateless Router Tier (Ingress and Connection Handling):

   • Upstream publishers connect to Pub/Sub via globally routed Anycast IP addresses using high-performance gRPC or REST over HTTP/2 (pubsub.googleapis.com).
   • Stateless router servers terminate client TLS connections, authenticate identity credentials via Cloud IAM, validate schema definitions (Avro or Protocol Buffers), and partition incoming message traffic.
   • Because routers are entirely stateless, any router can handle requests from any publisher, eliminating hot-spotting at the connection tier and enabling instantaneous horizontal auto-scaling to millions of requests per second.

2. Storage and Forwarder Tier (Persistence and Delivery):

   • Once validated by a router, messages are transferred to forwarder servers and persisted to Google's globally distributed storage layer, backed by Colossus (Google's next-generation distributed file system) and distributed consensus metadata logs.
   • Multi-Zone Synchronous Durability Quorum: Before Pub/Sub issues a publish acknowledgement (ACK) back to the publisher client, the message payload is synchronously replicated across storage nodes in at least two independent datacenter zones within the configured Google Cloud region. If an entire zone experiences hardware destruction milliseconds after publication, the message is preserved without loss.
   • Message Storage Policies (allowedPersistenceRegions): Although Pub/Sub offers a single global endpoint, organizations subject to data sovereignty regulations (such as GDPR or HIPAA) can define strict storage policies on topics. This guarantees that message payloads are stored exclusively within explicitly permitted geographic boundaries (e.g., europe-west1 and europe-west4), regardless of where publishers and subscribers are located globally.

+----------------------------------------------------------------------------------+
|                       CLOUD PUBSUB INTERNAL ARCHITECTURE                        |
+----------------------------------------------------------------------------------+
|                                                                                  |
|  [ Publishers ] ──(Anycast gRPC)──> [ Stateless Ingress Routers ]                |
|                                              │                                   |
|                                     (Route & Partition)                          |
|                                              ▼                                   |
|                                  [ Forwarder Server Tier ]                       |
|                                              │                                   |
|                        ┌─────────────────────┴─────────────────────┐             |
|                        ▼                                           ▼             |
|            [ Zone A Storage Node ]                     [ Zone B Storage Node ]   |
|                   (Colossus)                                  (Colossus)         |
|                        │                                           │             |
|                        └───────────── Synchronous Quorum ──────────┘             |
|                                              │                                   |
|                                     (Publish ACK returned)                       |
|                                              ▼                                   |
|                                  [ Independent Subscriptions ]                   |
|                                  ├── Subscription 1 (Pull Engine)                |
|                                  └── Subscription 2 (Push Webhook)               |
+----------------------------------------------------------------------------------+

---

2. Topics, Subscriptions, and the Fan-Out Paradigm

Cloud Pub/Sub strictly decouples the publishing entity from the consuming entity through Topics and Subscriptions:

• Topic: A named resource to which publishers send messages. Publishers have zero knowledge of which applications consume the messages or how many consumers exist.
• Subscription: A named resource representing an independent stream of messages from a specific topic to be delivered to a consuming application. A subscription maintains its own cursor, tracking which messages have been delivered and acknowledged.

One-to-Many Fan-Out Architecture

A single Pub/Sub topic can be bound to multiple independent subscriptions. When a message is published to the topic, Pub/Sub ensures that a copy or reference of that message is delivered to every attached subscription:

                          ┌──> [ Subscription A: Dataflow ] ──> BigQuery Analytics
[ E-Commerce Topic ] ─────┼──> [ Subscription B: Cloud Run ] ──> Fraud Detection
                          └──> [ Subscription C: GCS Direct ] ──> Cold Compliance Lake

• Decoupled Consumer Isolation: If Subscription A experiences a consumer failure and falls 4 hours behind in message processing, Subscription B continues processing messages in real time with sub-100 millisecond latency. Consumer lag on one subscription has zero impact on the throughput, delivery latency, or cursor state of any other subscription.
• Competing Consumers Pattern: When multiple worker instances (e.g., 20 pods running on Google Kubernetes Engine) connect to a single subscription (Subscription A), Pub/Sub distributes incoming messages among those 20 workers. Each individual message is delivered to only one worker instance within that subscription. This allows consumer pools to scale out horizontally to match incoming throughput without requiring manual partition rebalancing.

Pub/Sub vs. Traditional Partitioned Brokers (Apache Kafka)

---

3. Pull Subscriptions: Lease Management, Ack Deadlines, and StreamingPull

In a Pull subscription, the subscriber application initiates requests to retrieve messages from Cloud Pub/Sub. Managing message leases and acknowledgement deadlines is one of the most heavily tested topics on the exam.

Acknowledgement Deadlines and Lease Mechanics

When Pub/Sub delivers a message to a pulling subscriber, it starts an internal timer called the acknowledgement deadline (ackDeadlineSeconds):

• Configurable Range: 10 seconds to 600 seconds (default: 10 seconds).
• The In-Flight Lease: While the timer is running, the message is considered leased to that subscriber and is hidden from other consumers on the same subscription.
• Acknowledgement (ack): If the subscriber successfully processes the message and sends an acknowledge RPC before the deadline expires, Pub/Sub permanently marks the message as acknowledged and evicts it from active delivery.
• Negative Acknowledgement (nack): If the subscriber encounters a transient processing error (e.g., database connection pool full), it can send a nack (or call modifyAckDeadline with ackDeadlineSeconds = 0). Pub/Sub immediately terminates the lease and makes the message available for redelivery to another worker.
• Deadline Expiration (Automatic Redelivery): If the worker crashes or processing takes longer than ackDeadlineSeconds without an acknowledgement or lease extension, Pub/Sub assumes the worker died. The lease expires, and Pub/Sub automatically redelivers the message to another available worker.

Exam Trap — Duplicate Processing from Low Ack Deadlines: If a subscriber takes 45 seconds to process a batch of records, but the subscription's ackDeadlineSeconds is set to 10 seconds, Pub/Sub will redeliver the exact same message every 10 seconds to different workers in the pool. This causes duplicate processing storms, resource exhaustion, and high database write contention, even though the application has not crashed. The solution is either to increase ackDeadlineSeconds or use client library automatic lease renewal.

Synchronous Pull vs. StreamingPull

1. Synchronous Pull (projects.subscriptions.pull RPC):

   • A standard request-response HTTP/gRPC call where the client requests a batch of messages (maxMessages = 100).
   • If no messages are available, the call blocks until a timeout occurs or returns an empty list.
   • Drawbacks: High network polling overhead, increased latency, and requires manual lease management. If processing exceeds ackDeadlineSeconds, the developer must manually call modifyAckDeadline in separate background threads.
   • Best Use Case: Low-volume batch jobs, scheduled Cloud Functions running on Cloud Scheduler, or administrative debugging scripts.

2. StreamingPull (projects.subscriptions.streamingPull bidirectional gRPC stream):

   • Establishes a persistent, full-duplex, bidirectional gRPC connection between the subscriber client and Pub/Sub forwarders.
   • Messages are streamed to the client with sub-10ms delivery latency as soon as they are committed to storage.
   • Built-in Flow Control: Subscribers declare limits on unacknowledged messages (maxOutstandingMessages and maxOutstandingBytes). Once these limits are reached, the Pub/Sub forwarder pauses streaming to that client until existing messages are acknowledged, preventing out-of-memory (OOM) crashes.
   • Automatic Lease Extension (Heartbeating): The official Google Cloud client libraries automatically maintain an asynchronous background thread that periodically sends modifyAckDeadline requests to Pub/Sub for all messages currently being processed. The library extends the lease up to a configured maximum duration (e.g., maxDurationPerLeaseExtension = 1 hour), ensuring that long-running tasks are not prematurely redelivered to other workers.
   • Best Use Case: All production high-throughput streaming pipelines, including Apache Beam on Cloud Dataflow and high-performance microservices.

---

4. Push Subscriptions: Webhooks, Serverless Endpoints, and Authentication

In a Push subscription, Cloud Pub/Sub acts as an HTTPS client, initiating HTTPS POST requests to deliver messages to a pre-configured webhook endpoint.

Push Delivery Mechanics

• Target Endpoints: Publicly accessible HTTPS URLs, including Cloud Run services, App Engine endpoints, Google Kubernetes Engine Ingress controllers, or third-party webhooks.
• Delivery Envelope: Pub/Sub wraps the message in a standardized JSON payload:

  {
    "message": {
      "data": "ZXhhbXBsZSBwYXlsb2Fk",
      "attributes": {
        "source": "pos_terminal",
        "store_id": "982"
      },
      "messageId": "10482910482910",
      "publishTime": "2026-09-15T12:00:00.000Z"
    },
    "subscription": "projects/my-prod/subscriptions/orders-push-sub"
  }
  

• Acknowledgement via HTTP Status Codes:
  • Success (200, 201, 202, 204): Pub/Sub considers the message successfully delivered and marks it acknowledged.
  • Failure / Retry (Any other code, e.g., 500, 503, or HTTP timeout): Pub/Sub considers delivery failed and retries delivery using exponential backoff (configurable minimum and maximum retry delays, ranging from 10 seconds to 600 seconds).

OIDC Authentication for Secure Serverless Webhooks

Production push endpoints (such as Cloud Run or Cloud Functions) must never be left unauthenticated (allUsers). To deliver messages securely to private endpoints, Cloud Pub/Sub uses OpenID Connect (OIDC) authentication:

[ Cloud Pub/Sub ]
       │  1. Impersonates User-Managed Service Account
       ▼
[ Cloud IAM Credentials API ]
       │  2. Mints short-lived OIDC JSON Web Token (JWT) with Audience = Cloud Run URL
       ▼
[ Cloud Pub/Sub ]
       │  3. Sends HTTPS POST with Authorization: Bearer <OIDC_JWT>
       ▼
[ Cloud Run Service (Private) ]
          4. Validates JWT signature, expiration, and audience claim
          5. Executes business logic and returns HTTP 200 OK

Required IAM Configuration

1. Create a User-Managed Service Account: Dedicated identity for push delivery (e.g., pubsub-push-invoker@my-project.iam.gserviceaccount.com).
2. Grant Invoker Role to Service Account: Assign roles/run.invoker on the target Cloud Run service to the push service account.
3. Grant Token Creator to Pub/Sub Service Agent: The Google-managed Pub/Sub service agent (service-[PROJECT_NUMBER]@gcp-sa-pubsub.iam.gserviceaccount.com) must be granted the roles/iam.serviceAccountTokenCreator role on the push service account. This allows Pub/Sub to mint OIDC tokens on behalf of the service account.
4. Configure Subscription with OIDC: In the push subscription settings, enable authentication, select the service account, and set the Audience (aud) to the target service's base HTTPS URL.

---

5. Comparative Evaluation: Pull vs. StreamingPull vs. Push Subscriptions

---

6. Complete Message Lifecycle and Durability Guarantees

A Pub/Sub message transitions through seven distinct lifecycle phases from creation to eventual eviction:

+-----------------------------------------------------------------------------------+
|                         PUBSUB MESSAGE LIFECYCLE PHASES                           |
+-----------------------------------------------------------------------------------+
|  1. PUBLISH           Publisher transmits payload and attributes to topic         |
|                            │                                                      |
|  2. ROUTE & INGRESS   Ingress router validates schema, sets messageId & timestamp |
|                            │                                                      |
|  3. SYNCHRONOUS REPL  Replicated across >=2 zones in region; Publish ACK returned |
|                            │                                                      |
|  4. FAN-OUT QUEUE     Forwarders attach message reference to each subscription    |
|                            │                                                      |
|  5. DELIVERY & LEASE  Delivered to consumer; ackDeadlineSeconds timer starts      |
|                            │                                                      |
|             ┌──────────────┴──────────────┐                                       |
|             ▼                             ▼                                       |
|  6a. ACKNOWLEDGED           6b. EXPIRED / NACKED                                  |
|      Subscriber sends ack;      Lease expires or nack sent;                       |
|      evicted from delivery.     redelivered to alternate worker.                  |
|             │                                                                     |
|  7. RETENTION / EVICTION                                                          |
|      If retainAckedMessages=true, preserved until messageRetentionDuration.       |
+-----------------------------------------------------------------------------------+

Message Retention Boundaries

• Unacknowledged Message Retention: Cloud Pub/Sub retains all unacknowledged messages for a configurable period between 10 minutes and 7 days (default: 7 days). If a subscriber is completely offline for 6 days, no messages are lost; once the subscriber restarts, it resumes consuming the backlogged messages.
• Acknowledged Message Retention (retainAckedMessages): By default, when a message is acknowledged, Pub/Sub immediately marks it for deletion. However, subscriptions can enable retainAckedMessages = true and configure a retention duration up to 7 days. This preserves acknowledged messages in storage, unlocking the ability to replay historical streams.
• Topic-Level Retention: Retention can also be configured at the topic level (messageRetentionDuration), ensuring that all attached subscriptions inherit the ability to inspect historical data regardless of when each individual subscription was provisioned.

---

7. Message Replay: Seek by Timestamp and Subscription Snapshots

A frequent requirement in enterprise streaming systems is recovering from downstream failures. For example, if a deployment bug causes a downstream service to corrupt database records for two hours before being rolled back, how do you reprocess the original messages without re-publishing them from upstream producers?

Pub/Sub solves this through the Seek operation, which resets the message consumption cursor of a subscription:

1. Seek by Timestamp

• An operator or automated script rewinds the subscription cursor to any specific UTC timestamp within the retention window:

  gcloud pubsub subscriptions seek analytics-sub \
      --time="2026-09-15T10:00:00Z"
  

• Mechanics: Pub/Sub inspects all messages published after the specified timestamp. Any messages that were previously acknowledged are reset back to unacknowledged status. Subscribed consumers immediately begin re-receiving and processing those historical messages in chronological order.

2. Seek by Subscription Snapshot

• A Snapshot captures the point-in-time acknowledgement state of a subscription at a specific instant:

  # Create a point-in-time snapshot prior to deploying application update
  gcloud pubsub snapshots create pre-deploy-snap \
      --subscription=analytics-sub
  

• Snapshots retain unacknowledged messages for up to 7 days from the moment of snapshot creation, even if the subscription's standard retention expires.
• If a production deployment fails, the team can instantly revert the subscription to its exact pre-deployment state:

  gcloud pubsub subscriptions seek analytics-sub \
      --snapshot=pre-deploy-snap
  

• Snapshots can also be used to initialize new subscriptions with the exact cursor state of an existing subscription, enabling parallel pipeline testing against production traffic.

---

8. Realistic Exam Scenarios & Architecture Pitfalls