11.1 Event-Driven Architectures with Cloud Pub/Sub & Eventarc

Event-Driven Architectures with Cloud Pub/Sub & Eventarc

Core Principle: Modern enterprise cloud architectures rely on asynchronous, decoupled event distribution to absorb traffic spikes, decouple microservice dependencies, and enable real-time analytical streaming. Google Cloud provides two foundational services for event-driven systems: Cloud Pub/Sub, a globally distributed, horizontally scalable messaging middleware designed for high-throughput streaming and data buffering, and Eventarc, an event routing platform that captures and transforms control-plane and data-plane state changes from across 130+ Google Cloud services into standardized CloudEvents.

---

Cloud Pub/Sub Architecture & Core Topologies

Cloud Pub/Sub separates message producers (Publishers) from message consumers (Subscribers) through an asynchronous publish/subscribe pattern. It runs across all Google Cloud regions globally without requiring regional cluster provisioning, sharding, or capacity pre-allocation.

+-----------------------------------------------------------------------------------------+
|                              CLOUD PUB/SUB ARCHITECTURE MODEL                           |
+-----------------------------------------------------------------------------------------+
|  PUBLISHERS        |  App Engine / Cloud Run / GKE / On-Prem / IoT Devices              |
|                    |  - Publishes messages to a Topic with optional Ordering Keys       |
+--------------------+--------------------------------------------------------------------+
|  TOPICS & ROUTING  |  Global Topic Namespace (e.g., projects/my-p/topics/telemetry-raw) |
|                    |  - Automatic horizontal sharding and geo-distributed replication   |
+--------------------+--------------------------------------------------------------------+
|  SUBSCRIPTIONS     |  1-to-Many Fanout: Each subscription receives a copy of all msgs   |
|                    |  - Pull / Push / BigQuery Direct / Cloud Storage Direct            |
+--------------------+--------------------------------------------------------------------+
|  CONSUMERS         |  Dataflow / Cloud Run / BigQuery / GCS / External Microservices    |
+-----------------------------------------------------------------------------------------+

Core Entities & Message Anatomy

• Message: The atomic unit of transmission. It contains a binary payload (data up to 10 MB), user-defined key-value metadata (attributes), a system-generated globally unique identifier (messageId), and a server timestamp (publishTime).
• Topic: A named resource to which publishers send messages. Topics are global resources within a Google Cloud project.
• Subscription: A named resource representing a stream of messages from a single specific topic to be delivered to the subscribing application. Multiple subscriptions attached to a single topic receive independent copies of every published message (1-to-many fanout). Multiple workers pulling from a single subscription load-balance message consumption (competing consumers pattern).

Subscription Types & Ingestion Mechanics

DIRECT SINK SUBSCRIPTIONS (ZERO-CODE INGESTION)

                  +---> [ Pull Subscription ] --------> [ Dataflow Pipeline ]
                  |
                  +---> [ Push Subscription ] --------> [ Cloud Run Webhook ]
[ Pub/Sub Topic ]-+
                  +---> [ BigQuery Subscription ] ----> [ BigQuery Raw Table ] (Direct Write)
                  |
                  +---> [ Cloud Storage Sub ] --------> [ GCS Archive Bucket ] (Batch Files)

[!TIP] Direct BigQuery & Cloud Storage Subscriptions: Prior to the introduction of direct BigQuery and Cloud Storage subscriptions, architects had to deploy intermediate Cloud Functions or Dataflow pipelines solely to stream raw Pub/Sub messages into analytical storage. For pure ingestion pipelines where payload schema mapping is direct, using direct subscriptions eliminates compute operational overhead and reduces pipeline cost by over 60%.

---

Message Delivery Guarantees, Ordering & Reliability

Understanding Pub/Sub's distributed delivery semantics is critical for designing fault-tolerant distributed systems.

1. Delivery Guarantees & Acknowledgments

• At-Least-Once Delivery: By default, Pub/Sub guarantees that every published message is delivered at least once to each subscription. Because Pub/Sub is a distributed system, network timeouts or transient worker failures can cause a message to be redelivered. Consumer applications must be idempotent.
• Acknowledgment Deadline (ackDeadlineSeconds): When a subscriber receives a message, it has a configurable window (10 to 600 seconds, default 10s) to acknowledge (ack) the message. If the subscriber crashes or fails to ack/nack within the deadline, Pub/Sub marks the message as unacknowledged and redelivers it.
• Deadline Extension (modifyAckDeadline): Pub/Sub client libraries automatically extend the acknowledgment deadline in the background (lease management) while the worker is actively processing the message.

2. Strict Per-Key Ordering (Ordering Keys)

While standard Pub/Sub delivers messages out of order across parallel subscribers to maximize throughput, systems often require strict First-In, First-Out (FIFO) processing for specific entities (e.g., financial ledger transactions per bank account, state updates per IoT device).

ORDERING KEYS MECHANICS

Publisher Message Stream: 
  [ Msg 1: Key=Account_A ] ---> Sent to Topic
  [ Msg 2: Key=Account_B ] ---> Sent to Topic
  [ Msg 3: Key=Account_A ] ---> Sent to Topic (Must execute AFTER Msg 1)

Pub/Sub Router:
  - Key Account_A -> Affinity Partition Worker 1 -> [ Msg 1 ] then [ Msg 3 ]
  - Key Account_B -> Affinity Partition Worker 2 -> [ Msg 2 ]

* If Msg 1 processing fails or nacks -> Msg 3 is BLOCKED until Msg 1 succeeds or is purged to DLQ.

• Ordering Key Configuration: The publisher assigns an orderingKey string (e.g., account_10492) to the message metadata, and the subscription has message ordering enabled.
• Sequential Guarantees: Pub/Sub guarantees that messages sharing the same orderingKey are delivered to subscribers strictly in the order they were published.
• Head-of-Line Blocking Trap: If an ordered message fails processing or exceeds its ack deadline, Pub/Sub withholds all subsequent messages sharing that identical orderingKey until the failed message is successfully acknowledged, nacked, or redirected to a Dead-Letter Topic. Other ordering keys continue processing concurrently without impact.

3. Dead-Letter Topics (DLQ) & Retry Policies

When malformed payloads ("poison pills") cause continuous consumer crashes, unhandled messages cycle indefinitely through redelivery, consuming resources and stalling ordering keys.

DEAD-LETTER TOPIC (DLQ) & EXPONENTIAL BACKOFF

[ Raw Message ] ---> [ Worker Consumer ] ---> CRASH / NACK
                             |
                      Retry Attempt 1 (Min Backoff: 10s)
                             |
                      Retry Attempt 2 (Backoff: 20s)
                             |
                      Retry Attempt N (Exceeds MaxDeliveryAttempts = 5)
                             |
                             v
               [ Dead-Letter Topic (DLQ) ] ---> Alert / Debug Storage

• Dead-Letter Topics: Configured on a subscription with a maxDeliveryAttempts threshold (between 5 and 100). When a message fails delivery more than $N$ times, Pub/Sub automatically routes the message to a designated Dead-Letter Topic and acknowledges it on the primary subscription.
• Retry Policies:
  • Immediate Retry: Pub/Sub attempts redelivery immediately upon ack deadline expiration or explicit nack.
  • Exponential Backoff: Configured with a minimumBackoff (e.g., 10s) and maximumBackoff (e.g., 600s). Pub/Sub doubles the delay between successive delivery attempts, preventing downstream database thundering herd problems during outages.

4. Message Retention & Seek / Replay

• Retention Limits: Pub/Sub retains unacknowledged messages on subscriptions for up to 31 days (default 7 days). Topics can also be configured with message retention independent of subscriptions.
• Retaining Acknowledged Messages: Subscriptions can be configured to retain acknowledged messages within the retention window.
• Seek Operation: Allows operators to rewind a subscription's message consumption cursor back to a specific timestamp in the past or to a named snapshot. This enables:
  • Replaying historical events after deploying a bug fix in consumer code.
  • Backfilling a newly created analytics pipeline with past events.
  • Purging accumulated backlogs instantly by seeking to the current timestamp.

---

Pub/Sub Standard vs. Pub/Sub Lite

Google Cloud provides two distinct tiers of the Pub/Sub messaging service tailored to different architectural and economic requirements.

---

Eventarc: Unified CloudEvents Routing

While Pub/Sub is engineered for high-volume data streaming, Eventarc is engineered for event-driven orchestration and state change notifications across Google Cloud infrastructure.

+-----------------------------------------------------------------------------------------+
|                               EVENTARC EVENT ROUTING MODEL                              |
+-----------------------------------------------------------------------------------------+
|  EVENT SOURCES (130+ Google Cloud Services)                                             |
|  1. Direct Events: Cloud Storage (object finalize), Pub/Sub, Firebase                  |
|  2. Cloud Audit Logs: Any GCP API mutation (Compute VM created, BigQuery job, IAM edit) |
|  3. Custom Events: External applications via Eventarc Publishing API                    |
+-----------------------------------------------------------------------------------------+
|                                            |                                            |
|                                            v                                            |
|  EVENTARC ROUTING ENGINE (Filters on attributes: type, serviceName, methodName)        |
|  - Standardizes all payloads into CloudEvents v1.0 specification                        |
+-----------------------------------------------------------------------------------------+
|                                            |                                            |
|                                            v                                            |
|  EVENT DESTINATIONS                                                                     |
|  - Cloud Run Services / Cloud Functions (2nd Gen) / GKE Services / Cloud Workflows      |
+-----------------------------------------------------------------------------------------+

Key Eventarc Architectural Capabilities

1. CloudEvents Standard Format: Eventarc encapsulates all events in the CNCF CloudEvents v1.0 open standard specification, providing consistent metadata attributes (id, source, type, time, datacontenttype, and data). This prevents proprietary vendor lock-in.
2. Cloud Audit Log Integration: Eventarc can capture events from any Google Cloud service that writes to Cloud Audit Logs (over 130+ GCP services). For example, creating a new Compute Engine VM, altering an IAM policy, or completing a Spanner backup can trigger an automated remediation workflow without polling or custom application code.
3. Eventarc Triggers & Filtering: Triggers define declarative routing rules matching specific attributes (e.g., type: google.cloud.audit.log.v1.written, serviceName: storage.googleapis.com, methodName: storage.objects.create).
4. Secure Delivery: Eventarc invokes destinations using authenticated HTTPS calls, automatically managing IAM service account OIDC identity tokens.

Pub/Sub vs. Eventarc Decision Matrix

---

Concrete Architectural Scenario: Real-Time E-Commerce Ledger & Fraud Detection

Scenario Profile

• Client: Global E-Commerce Retailer handling 50,000 transactions per second during flash sales.
• Requirements:
  1. Financial ledger updates for each user account must process strictly in chronological order without race conditions.
  2. Malformed transactions must not block legitimate customer orders.
  3. Raw transactions must stream into BigQuery for real-time analytics with zero custom ETL compute infrastructure.
  4. Any administrative modification to payment IAM service accounts must immediately trigger an automated security audit function.

[ Web/Mobile Checkout ] ---> (Publishes with OrderingKey=Account_ID) ---> [ Pub/Sub Topic: Orders ]
                                                                                 |
                                +------------------------------------------------+----------------+
                                |                                                |                |
                                v                                                v                v
                [ Subscription: Ledger-Worker ]                   [ Subscription: Direct BQ ]  [ Subscription: GCS ]
                - Message Ordering Enabled                        - Direct BigQuery Stream     - Avro Raw Archive
                - DLQ Topic: orders-dlq (maxAttempts=5)           - Zero Compute Footprint     - Coldline Tier
                - Exponential Backoff: 5s to 300s                                |
                                |                                                v
                                v                                    [ BigQuery Real-Time Table ]
                     [ Cloud Run Ledger Worker ]

[ IAM Admin Change ] ---> [ Cloud Audit Log ] ---> [ Eventarc Trigger ] ---> [ Cloud Run Security Auditor ]

Architecture Blueprint

1. Sequential Order Processing: The checkout application publishes order messages with orderingKey = user_account_id. The Ledger-Worker subscription has message ordering enabled, guaranteeing strict chronological processing per customer account.
2. Fault Isolation: The subscription configures a Dead-Letter Topic (orders-dlq) with maxDeliveryAttempts = 5. If an unparseable order payload fails 5 times, it is automatically routed to the DLQ, unblocking the ordering key for subsequent customer transactions.
3. Serverless Analytical Ingestion: A BigQuery Subscription streams orders directly into the analytics warehouse with zero intermediate Cloud Functions or Dataflow worker costs.
4. Security Governance: An Eventarc Trigger listens for google.cloud.audit.log.v1.written events on the iam.googleapis.com service, routing administrative changes immediately to a Cloud Run Security Auditor service.

---

[!IMPORTANT] Exam Watch:

1. If an exam question asks for strict message ordering per customer/entity, select Pub/Sub Ordering Keys. Remember: ordering is per-key, not global across the entire topic.
2. If poison-pill messages are causing pipeline crashes or head-of-line blocking on ordered subscriptions, configure a Dead-Letter Topic (DLQ) with a retry policy.
3. If an exam scenario requires reacting to GCP service lifecycle events (such as Cloud Storage uploads, BigQuery table creation, or IAM modifications) across 130+ Google Cloud services using standardized formats, Eventarc is the correct architectural choice.