3.2 High Availability, Disaster Recovery Patterns & RTO/RPO

High Availability (HA) vs. Disaster Recovery (DR)

In the Google Cloud Well-Architected Framework, High Availability and Disaster Recovery are complementary yet distinct engineering disciplines:

• High Availability (HA) focuses on eliminating Single Points of Failure (SPOFs) within normal operating conditions, ensuring the system remains operational during localized hardware, rack, network, or single-zone disruptions.
• Disaster Recovery (DR) focuses on business continuity during catastrophic, rare events that incapacitate an entire geographic region (e.g., major natural disasters, fiber cuts, regional grid failures) or widespread logical data corruption.

Quantifying Availability Metrics (The "Nines")

Availability is expressed mathematically as the percentage of operational uptime over a rolling calendar period:

$\text{Availability} = \frac{\text{Total Uptime}}{\text{Total Uptime} + \text{Total Downtime}} \times 100\%$

---

Fault Domains: Zonal vs. Regional Resilience

Google Cloud partitions global infrastructure into strict fault isolation boundaries:

Google Cloud Global Backbone
  ├── Region: us-central1 (Iowa)
  │     ├── Zone: us-central1-a (Isolated power, cooling, compute fabric)
  │     ├── Zone: us-central1-b (Isolated power, cooling, compute fabric)
  │     └── Zone: us-central1-c (Isolated power, cooling, compute fabric)
  └── Region: europe-west1 (Belgium)
        ├── Zone: europe-west1-b
        ├── Zone: europe-west1-c
        └── Zone: europe-west1-d

Zonal Resilience Patterns

• Regional Managed Instance Groups (Regional MIGs): Automatically spread instances evenly across three zones in a region. If us-central1-a fails, the regional MIG automatically provisions replacement instances in us-central1-b and us-central1-c without human intervention.
• Regional Persistent Disks (Regional PD): Synchronously replicate block storage data across two zones in the same region. If a database VM crashes along with its host zone, the standby VM in the second zone can immediately attach the Regional PD and resume service with zero storage data loss.

Regional Resilience Patterns

• Cloud Storage Multi-Region / Dual-Region: Objects written to a Dual-Region bucket (e.g., nam4 for Iowa/South Carolina) are geo-redundantly stored across distinct regions separated by hundreds of miles. Google Cloud offers Turbo Replication, providing a 100% replication SLA within 15 minutes for dual-region buckets.
• Cloud Spanner Multi-Region: Executes distributed transactions across multiple regions using atomic clock-synchronized TrueTime and Paxos consensus, providing 99.999% (Five Nines) availability with synchronous replication and zero RPO.

---

The Disaster Recovery Spectrum

Organizations select a DR topology based on the balance between financial cost and downtime tolerance. The DR continuum spans four canonical architectures:

◄ Lower Cost / Slower Recovery                       Higher Cost / Instant Recovery ►

┌─────────────────┐   ┌─────────────────┐   ┌─────────────────┐   ┌─────────────────┐
│ Backup & Restore│   │   Pilot Light   │   │  Warm Standby   │   │  Hot Standby    │
│ (Cold DR)       │   │  (Cold Standby) │   │ (Scaled Mirror) │   │ (Active-Active) │
│                 │   │                 │   │                 │   │                 │
│ RTO: Hours/Days │   │ RTO: Tens of min│   │ RTO: Minutes    │   │ RTO: Near-Zero  │
│ RPO: Hours      │   │ RPO: Seconds/Min│   │ RPO: Seconds    │   │ RPO: Zero (0)   │
└─────────────────┘   └─────────────────┘   └─────────────────┘   └─────────────────┘

1. Backup and Restore (Cold DR)

• Architecture: Data and database dumps are backed up periodically (e.g., hourly snapshots or daily exports) and shipped to Cloud Storage in a secondary region. Compute resources in the recovery region are zero.
• Failover Process: In a disaster, operations teams provision new VPC networks, spin up compute instances via Terraform, and restore database state from storage snapshots.
• Trade-offs: Lowest cost; highest RTO (hours to days) and highest RPO (hours).

2. Cold Standby (Pilot Light)

• Architecture: Core database data is continuously replicated asynchronously to a secondary region (e.g., a cross-region Cloud SQL read replica). Core compute and network infrastructure definitions (Terraform / Golden Machine Images) exist in the secondary region, but VM instances are turned off or kept at a count of zero.
• Failover Process: When disaster strikes, automation scripts promote the database replica to become the standalone primary database, and the secondary Managed Instance Group scales from 0 to target capacity.
• Trade-offs: Low continuous cost; RTO of 15 to 45 minutes; RPO of seconds to minutes.

3. Warm Standby

• Architecture: A fully functional, scaled-down version of the production environment runs 24/7 in the secondary region. The secondary database is actively replicating, and the secondary MIG runs at minimal capacity (e.g., 20% of production traffic capacity).
• Failover Process: The load balancer redirects traffic to the secondary region, and the secondary MIG autoscales out rapidly to 100% capacity.
• Trade-offs: Medium-high cost; RTO of minutes; RPO of seconds.

4. Hot Standby / Multi-Region Active-Active

• Architecture: Production traffic is simultaneously served by fully provisioned infrastructure in two or more regions. An External Application Load Balancer routes users to the nearest healthy region. Underlying databases leverage multi-region synchronous replication (Cloud Spanner) or active-active multi-master replication.
• Failover Process: Completely automated and instantaneous. If Region A fails health checks, Anycast routing instantly sends 100% of global traffic to Region B.
• Trade-offs: Highest infrastructure cost; RTO ≈ 0; RPO = 0.

---

Calculating and Designing for RTO and RPO

Two fundamental metrics govern Disaster Recovery engineering:

                       DISASTER OCCURS
                             │
◄── RPO Window ──────────────┼────────────── RTO Window ──►
[Last Valid Data State]      │           [Full Service Restored]
                             ▼
                     (Lost Transactions)

• Recovery Point Objective (RPO): The maximum acceptable data loss measured in time. It defines the delta between the last successfully committed/replicated transaction and the point of disaster. An RPO of 15 minutes means the business can tolerate losing up to 15 minutes of transactional data.
• Recovery Time Objective (RTO): The maximum acceptable duration of system downtime. It defines how quickly the system must be restored to full production operation after an outage begins.

$\text{Total Outage Impact} = \text{RTO (Downtime Duration)} + \text{Data Reconstitution Time from RPO}$

Cross-Region Data Replication Trade-Offs

Designing data persistence across geographic regions involves an immutable physical trade-off between consistency, latency, and disaster resilience:

Replication Spectrum:
Synchronous (Paxos / TrueTime)       Asynchronous (Log Shipping)
├────────────────────────────────────┼────────────────────────────────────┤
• RPO = 0 (Guaranteed Zero Data Loss) • RPO > 0 (Replication Lag Delta)
• Strong Consistency Globally         • Eventual Consistency
• Higher Write Latency (50-150ms RTT) • Low Write Latency (<5ms Local Write)
• Cloud Spanner Multi-Region          • Cloud SQL Cross-Region Replicas

1. Synchronous Replication

• Services: Cloud Spanner Multi-Region, Cloud Storage Dual/Multi-Region.
• Mechanism: A write transaction is not acknowledged to the client until a quorum of distributed voting replicas across distinct regions successfully commit the transaction via Paxos consensus.
• Advantages: RPO = 0. In the event of a catastrophic regional failure, zero data is lost, and secondary regions can continue serving writes immediately without data reconciliation.
• Trade-off: Elevated write latency due to cross-region network round-trip time (RTT) governed by the speed of light in fiber optics (typically 30-100ms).

2. Asynchronous Replication

• Services: Cloud SQL Cross-Region Read Replicas, AlloyDB Secondary Clusters, Bigtable Multi-Cluster Routing.
• Mechanism: Writes commit locally to the primary database instance with sub-5ms latency. Transaction write-ahead logs (WAL) are shipped asynchronously over the Google private network to read replicas in secondary regions.
• Advantages: Extremely low local write latency; primary transactions are decoupled from remote network delays.
• Trade-off: Non-zero RPO. Replication lag represents the real-time difference between the primary database and the secondary replica. If the primary region collapses abruptly, un-replicated transactions in flight are permanently lost upon replica promotion.

[!IMPORTANT] Replica Promotion Rule: Promoting a Cloud SQL cross-region replica to become a primary standalone database is a one-way, irreversible operation. Once promoted, the replica severs replication ties with the original master. If the original primary region later recovers, architects must establish a brand-new replication stream to prevent split-brain data corruption.

---

Failover Orchestration and DNS Failover

When a primary region fails, redirecting user traffic to the secondary disaster recovery site requires deterministic traffic orchestration:

Traffic Steering Approaches:

1. Global Load Balancing (Anycast VIP) ──> [Instant Sub-Second Rerouting]
   Client ──> 34.120.X.X ──[Google Edge Proxy]──> Healthy Surviving Region

2. Cloud DNS Health-Checked Failover  ──> [Delayed by Client DNS TTL Caching]
   Client ──> Queries DNS ──[TTL Delay 60-300s]──> Updates to Secondary IP

1. Global External Load Balancer Anycast Failover (Recommended)

• Google Cloud Global External Application Load Balancers advertise a single Anycast IP address from over 180+ global Edge Points of Presence.
• The load balancer continuously evaluates regional backend health checks. If all backend VMs in us-central1 fail health checks, Google's edge infrastructure automatically reroutes traffic to backends in europe-west1 in under 5 seconds.
• Architectural Advantage: Zero dependency on client-side DNS cache updates or TTL expirations.

2. Cloud DNS Routing Policies

• Failover Policy: Primary and backup geo-targets configured with health checks. If the primary endpoint fails health checks, Cloud DNS automatically updates responses to return the backup IP.
• Limitation: Client operating systems, ISP recursive resolvers, and intermediate proxy servers frequently ignore or cache DNS TTL values (e.g., retaining cached IPs for 300 to 3,600 seconds). DNS failover is inherently slower than Anycast load balancing and introduces delayed RTO.