3.1 Scalability, Elasticity & Capacity Planning

Architectural Foundations of Scalability and Elasticity

In enterprise cloud system design, scalability and elasticity address two distinct dimensions of capacity management:

• Scalability refers to the structural ability of a system to handle increasing load by adding resources without redesigning the core architecture.
• Elasticity refers to the dynamic, automated adaptation of capacity in real-time, provisioning resources when demand surges and deprovisioning them when demand subsides to optimize operational cost.

For the Google Cloud Professional Cloud Architect exam, you must evaluate whether an application's statefulness, dependencies, and traffic patterns dictate a vertical or horizontal scaling strategy, and configure the appropriate autoscaling mechanisms to prevent degradation during abrupt load shifts.

---

Vertical vs. Horizontal Scaling Patterns

Architects categorize compute expansion into two classical paradigms:

Vertical Scaling (Scale-Up)         Horizontal Scaling (Scale-Out)
┌─────────────────────────┐          ┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐
│      Compute Node       │          │ Node │ │ Node │ │ Node │ │ Node │
│  (32 vCPU -> 128 vCPU)  │          │  #1  │ │  #2  │ │  #3  │ │  #4  │
│  (128 GB  -> 512 GB)    │          └──────┘ └──────┘ └──────┘ └──────┘
└─────────────────────────┘          ▲ Dynamic Load Balancer Dispatch ▲

1. Vertical Scaling (Scale-Up / Scale-Down)

Vertical scaling increases or decreases the physical or virtual capacity of an individual machine (such as moving a Compute Engine instance from an e2-standard-4 to an n2-standard-32, or upgrading a Cloud SQL instance to 96 vCPUs).

• Architectural Implications: Simpler to implement for monolithic legacy workloads that maintain state in local memory or local file systems. No distributed concurrency or network partitioning logic is required.
• Limitations: Hard hardware boundaries (hypervisor limits on maximum sockets and memory channels), prohibitive exponential cost at the high end of machine families, and unavoidable downtime during instance resizing (which requires a VM stop/start cycle).

2. Horizontal Scaling (Scale-Out / Scale-In)

Horizontal scaling adjusts the number of discrete, uniform compute nodes running in parallel behind an intelligent distribution layer (such as an External Application Load Balancer or Internal TCP/UDP Load Balancer).

• Architectural Implications: Enables virtually unbounded elasticity. Treats infrastructure as disposable units ("cattle, not pets").
• Prerequisites: Requires stateless application tiers. All persistent session state, authentication tokens, and transaction contexts must be externalized into distributed storage tiers (e.g., Memorystore for Redis, Cloud Spanner, Cloud SQL, or Firestore).

---

Autoscaling Policies in Managed Instance Groups (MIGs)

Compute Engine Managed Instance Groups (MIGs) automate horizontal elasticity for VM-based workloads. An autoscaler dynamically adds or removes instances based on measured telemetry relative to a defined target. Choosing the incorrect autoscaling metric is one of the most common causes of production outages and poor exam scores.

Autoscaling Metric Trigger Hierarchy:
1. CPU Utilization         ──> Compute-bound jobs (video encoding, batch crypto)
2. Cloud LB Capacity       ──> Web services, REST APIs, public HTTP endpoints
3. Cloud Monitoring Custom ──> Application metrics (active websockets, thread queue)
4. Pub/Sub Queue Backlog   ──> Asynchronous workers, event-driven pipelines

1. Target CPU Utilization

• Mechanism: The autoscaler maintains the average CPU utilization of all ready instances at or below a target percentage (e.g., 65%).
• Best For: Compute-intensive, synchronous workloads where CPU consumption directly correlates with user transaction volume (e.g., image rendering, algorithmic processing).
• Configuration Rule: Do not set the target CPU to 90%+. High target utilization leaves insufficient headroom for traffic spikes while new VMs are booting, causing request queuing and HTTP 504 Gateway Timeouts.

2. HTTP(S) Load Balancing Serving Capacity

• Mechanism: Calculates capacity based on Cloud Load Balancing telemetry. Can be configured as:
  • Target utilization: Fraction of maximum backend capacity (0.0 to 1.0).
  • Target RPS (Requests Per Second): Explicit request rate per instance.
• Best For: Web applications and HTTP REST APIs where requests may be I/O-bound (waiting on external database calls) rather than CPU-bound.
• Architectural Advantage: When regional backends reach 100% serving capacity, Global External Application Load Balancers seamlessly overflow excess requests to the nearest healthy alternate region with spare capacity.

3. Cloud Monitoring Custom Metrics

• Mechanism: Autoscale based on any standard or custom metric exported to Cloud Monitoring (e.g., custom.googleapis.com/app/active_connections or JVM heap utilization).
• Best For: Microservices whose bottlenecks are non-CPU resources (e.g., memory utilization, database connection pool exhaustion, open WebSocket channels).

4. Cloud Pub/Sub Queue Depth

• Mechanism: Tracks the backlog metric pubsub.googleapis.com/subscription/num_undelivered_messages.
• Formula: The autoscaler calculates the required number of VM instances by dividing the total unacknowledged message backlog by the processing capacity of an individual worker, scaled against the target processing window.
• Best For: Asynchronous background workers, ingestion pipelines, and event-driven architectures.

Required Instances = ⌈ Unacknowledged Messages in Queue / Target Messages Processed per VM ⌉

[!WARNING] Cool-down (Initialization) Period Gotcha: When a new instance launches, it requires time to boot the OS, initialize Docker containers, and warm up caches. The cool-down period instructs the autoscaler to ignore metrics from newly created instances during this startup window (e.g., 120 seconds). If set too short, the autoscaler observes high load on the struggling cluster and aggressively over-provisions unnecessary VMs (flapping/thrashing).

---

Quota and Rate Limit Management

Elasticity is strictly bounded by Google Cloud resource quotas. A cloud architecture designed to scale to 100 instances will fail during an incident if project quotas limit provisioning to 24 vCPUs.

Quota Dimensions

1. Allocation Quotas: Maximum quantity of physical resources that can be provisioned at any given time (e.g., Regional vCPUs (all regions), N2 CPUs in us-central1, Standard Persistent Disk (GB) in europe-west1, In-use IP addresses).
2. Rate Quotas (API Rate Limits): Maximum frequency of administrative API requests (e.g., Compute Engine API queries per minute per user or Cloud KMS cryptoKeyVersions.create per minute).

Hierarchical Scope

• Zonal Quotas: Constrain resources inside a specific zone (e.g., specialized GPU types such as NVIDIA H100s in us-central1-a).
• Regional Quotas: Constrain regional aggregates (e.g., total vCPUs across all zones in us-central1).
• Project-Level Quotas: Enforce organizational and billing guardrails at the project boundary.

Organization Root
  └── Project A (Regional vCPU Quota: 1000 vCPUs)
        ├── us-central1-a (Consuming 300 vCPUs)
        ├── us-central1-b (Consuming 400 vCPUs)
        └── us-central1-c (Consuming 300 vCPUs) ──> [CAPACITY MAX REACHED]

Proactive Quota Architecture

• Monitoring & Alerting: Configure Cloud Monitoring alert policies on serviceruntime.googleapis.com/quota/allocation/usage divided by quota/allocation/limit. Establish threshold alerts at 80% usage.
• Lead Time for Quota Increases: Quota increases for standard resources in popular regions are evaluated automatically via machine learning; however, large increases, specialized compute (GPUs/TPUs), or new projects require human review and can take several business days. Architects must request quota increases weeks prior to major seasonal events.

---

Capacity Planning: Predictable vs. Spiky Workloads

Managing Abrupt Traffic Spikes

Reactive autoscaling alone cannot handle sudden 10x traffic surges within 30 seconds, because Compute Engine VMs require 60 to 120 seconds to boot, register with the load balancer, and pass initial health checks. To mitigate this:

1. Pre-warming: Scale up the minimum instance count (minNumReplicas) 30 minutes prior to a scheduled event.
2. On-Demand Capacity Reservations: Reserve specific compute instance types in specific zones (e.g., n2-standard-16 in us-east4-a) to guarantee hardware availability during regional peak demand periods, preventing ZONE_RESOURCE_POOL_EXHAUSTED errors.

---

Decoupling Architectures Using Queues and Buffers

When upstream traffic surges exceed the capacity of downstream transactional systems (such as a legacy database or external payment gateway), synchronous HTTP calls result in thread exhaustion, timeouts, and cascading system failure. Asynchronous decoupling introduces an elastic intermediary buffer.

Synchronous Anti-Pattern (Brittle under load):
Client ──[HTTP Request]──> Web App ──[Synchronous Write]──> Database (Exhausts Connection Pool / Crashes)

Decoupled Architectural Pattern (Resilient & Elastic):
Client ──[HTTP Request]──> Web App ──[Publish]──> Cloud Pub/Sub ──[Buffer]──> Worker MIG ──[Batched Write]──> Database
                                                        │
                                                        └──[Failed Messages]──> Dead-Letter Topic (DLQ)

1. Cloud Pub/Sub vs. Cloud Tasks

• Cloud Pub/Sub: High-throughput, many-to-many event streaming service. Automatically scales to millions of messages per second with global ingress. Ideal for event-driven ingestion, analytics pipelines, and broadcasting.
• Cloud Tasks: Point-to-point, asynchronous task dispatch. Provides fine-grained rate limiting (e.g., maximum 50 tasks/second dispatched to a backend), explicit execution scheduling, task deduplication, and customized retry policies. Ideal for worker queues interfacing with rate-limited downstream APIs.

2. Backpressure and Traffic Smoothing (Load Leveling)

By placing Cloud Pub/Sub in front of processing workers, the queue absorbs massive ingress spikes instantaneously without dropping transactions. Worker pools scale out independently based on queue depth, processing tasks at a sustainable rate that protects the underlying database.

3. Fault Resilience: Retries, Jitter & Dead-Letter Topics (DLQs)

• Exponential Backoff with Jitter: Consumer retry logic must add randomized jitter to exponential delay intervals. If 1,000 workers retry a failed database connection at exact fixed intervals (e.g., precisely every 2.0 seconds), they generate a "thundering herd" that prevents the database from recovering.
• Dead-Letter Topics (DLQ): When a "poison pill" message (e.g., malformed JSON payload) fails processing repeatedly, Pub/Sub forwards it to a Dead-Letter Topic after a configurable maxDeliveryAttempts threshold (e.g., 5 attempts). This unblocks the queue and allows healthy transactions to proceed while preserving failed payloads for post-mortem analysis.