10.2 Designing High Availability Architectures Across AWS Data Services

10.2 Designing High Availability Architectures Across AWS Data Services

High Availability (HA) in AWS data engineering refers to the capability of an analytics platform to remain continuously operational and accessible during hardware faults, software failures, or whole Availability Zone (AZ) outages. Unlike Disaster Recovery—which focuses on cross-Region recovery after catastrophic events—HA focuses on intra-Region fault tolerance, automatic health monitoring, and self-healing infrastructure.

---

1. High Availability Storage and Database Topologies

Achieving high availability at the data tier requires decoupling compute nodes from persistent storage while distributing state across redundant physical facilities (AZs).

Amazon Aurora Decoupled Quorum Architecture

Amazon Aurora achieves unmatched database availability through its custom distributed storage volume:

• 6-Way Replication Across 3 AZs: Aurora automatically replicates every 10 GB storage chunk six times across three AZs within a Region.
• Quorum Mechanics:
  • Write Quorum (4 of 6): A write operation is acknowledged as successful as soon as 4 out of the 6 storage nodes confirm the write. A complete loss of one entire AZ (2 copies) does not impair write availability.
  • Read Quorum (3 of 6): Read operations require agreement from 3 nodes, allowing rapid reconstruction of corrupted blocks.
• Storage Auto-Healing: Storage disk failures are repaired transparently in the background without affecting database compute availability.
• Read Replica Auto-Failover: If the primary writer instance fails, Aurora automatically promotes one of up to 15 Read Replicas to primary writer status in under 120 seconds (often within 30 seconds).

                    Aurora Shared Distributed Storage Volume
   +-----------------------------------------------------------------------+
   |  AZ 1: [ Data Copy 1 ]  [ Data Copy 2 ]                              |
   |  AZ 2: [ Data Copy 3 ]  [ Data Copy 4 ]   <-- Write Quorum: 4 of 6    |
   |  AZ 3: [ Data Copy 5 ]  [ Data Copy 6 ]   <-- Read Quorum:  3 of 6    |
   +-----------------------------------------------------------------------+

Amazon RDS Multi-AZ Deployment (Instance Level)

For traditional relational engines (PostgreSQL, MySQL, SQL Server), standard Amazon RDS Multi-AZ uses synchronous block-level storage replication:

• Provisions a primary database instance in AZ-A and a standby instance in AZ-B.
• Synchronously replicates storage updates at the EBS volume layer.
• In the event of primary hardware failure, AZ outage, or database crash, RDS automatically updates DNS records to point to the standby instance. Application connection strings remain unchanged.

---

2. High Availability in Distributed Analytics Engines

Distributed compute frameworks like Apache Spark and Hadoop on Amazon EMR require special configuration to prevent primary node failures from aborting long-running ETL jobs.

Amazon EMR Multi-Master High Availability

By default, a standard EMR cluster has one primary node, creating a single point of failure (SPOF). Choosing an EMR cluster with multiple primary nodes provisions 3 primary nodes within the cluster's single subnet and Availability Zone:

• HDFS NameNode HA: Active NameNode running on Primary Node 1 with Standby NameNodes on Nodes 2 and 3, coordinated via Apache ZooKeeper and Quorum Journal Managers.
• YARN ResourceManager HA: Active ResourceManager with automatic failover to standbys.
• Availability boundary: Instance fleets can consider subnets in multiple AZs when the cluster is launched, but EMR selects one subnet and provisions every node in that one AZ. Three primary nodes tolerate a primary-node failure, not an AZ failure. Use reproducible data in S3 and orchestration that can launch a replacement cluster in another AZ for broader recovery.

AWS Glue and Serverless Compute Fault Tolerance

AWS Glue ETL jobs are serverless and inherently resilient:

• Glue abstracts worker infrastructure, but architecture should not assume a documented worker-AZ topology or that every worker loss is transparent. Keep durable inputs in S3, configure retries, and make outputs idempotent so a failed attempt can be rerun safely.

---

3. Fault-Tolerant Streaming Data Pipelines

Real-time ingestion pipelines must ingest continuous streams without data loss during infrastructure failures.

Amazon MSK (Managed Streaming for Apache Kafka) HA Best Practices

To ensure multi-AZ fault tolerance in Amazon MSK:

1. Broker Distribution: Provision MSK clusters with brokers evenly distributed across 3 Availability Zones.
2. Topic Replication Factor: Set default.replication.factor = 3 so every Kafka topic partition has a primary leader broker in one AZ and follower replicas in two other AZs.
3. In-Sync Replicas Configuration: Set min.insync.replicas = 2 and require producers to use acks = all (or acks = -1). This guarantees that a write is only acknowledged after it has been durably committed to at least 2 brokers across distinct AZs.

Amazon Kinesis Data Streams Durability

Amazon Kinesis Data Streams natively provides multi-AZ high availability:

• Data records are synchronously replicated across 3 Availability Zones before write acknowledgment.
• Enhanced Fan-Out (EFO): Dedicated 2 MB/sec throughput per consumer per shard using HTTP/2 push delivery, shielding downstream consumers from reading contention during node recovery.
• Retention periods can be extended up to 365 days, providing a safety buffer to replay data after downstream pipeline crashes.

Stateful Streaming Checkpointing

Stateful stream processing using Managed Service for Apache Flink or Glue Streaming preserves state during worker node failure:

• Managed Service for Apache Flink coordinates checkpoints and service-managed snapshots; applications should not assume a user-visible immutable S3 checkpoint path.
• Embedded RocksDB can back large application state, depending on runtime and application configuration.
• After worker failure, Flink restores state and source positions from a completed checkpoint. End-to-end exactly-once results additionally require replayable sources plus sinks and application logic that support transactional or idempotent recovery.

---

4. Resilient Search and Query Infrastructure

Amazon OpenSearch Service Multi-AZ with Standby

For enterprise operational analytics, Amazon OpenSearch Service offers Multi-AZ with Standby:

• Node Allocation: Distributes active data nodes across 2 AZs and standby nodes in a 3rd AZ.
• Enforced Shard Distribution: Primary shards and replica shards are strictly isolated in separate AZs.
• Automated Failover: If an active node or AZ fails, OpenSearch promotes a standby node in under 1 minute without dropping indexing requests or incurring write performance degradation.