6.1 Amazon Redshift Architecture, RA3 Nodes & Managed Storage

6.1 Amazon Redshift Architecture, RA3 Nodes & Managed Storage

Data Warehousing & MPP Architecture Fundamentals

In modern data engineering architectures, enterprise data warehouses serve as the central repository for analytical processing (OLAP). Unlike transactional databases (OLTP) optimized for row-by-row CRUD operations and low-latency writes, analytical data warehouses are optimized for complex aggregations, multi-table joins, and scanning billions of records across narrow column subsets.

Amazon Redshift is AWS's flagship fully managed cloud data warehouse. It utilizes a Massively Parallel Processing (MPP), shared-nothing architecture. In an MPP database, multiple compute nodes work in parallel to execute single query tasks across partitioned datasets. Every compute node has dedicated CPU, RAM, and storage slices, ensuring that computational capacity scales linearly as cluster size increases.

---

Redshift Cluster Anatomy: Leader Node & Compute Nodes

An Amazon Redshift cluster consists of two distinct node tiers: a single Leader Node and one or more Compute Nodes.

Client App (JDBC/ODBC) ---> [ Leader Node ] ---> Compute Node 1 (Slice 0, Slice 1)
                                            ---> Compute Node 2 (Slice 2, Slice 3)

1. The Leader Node

The Leader Node acts as the SQL entry point and orchestrator for the entire cluster. Client applications connect exclusively to the Leader Node via standard PostgreSQL-compliant JDBC or ODBC drivers (typically on port 5439).

Key functions of the Leader Node include:

• Query Parsing & Optimization: Accepts SQL queries, parses execution trees, and optimizes execution plans.
• Code Compilation: Compiles optimized query plans into C++ execution code segments and distributes compiled binary code to Compute Nodes.
• Aggregating Partial Results: Receives intermediate query results returned by compute nodes, performs final aggregations or sorting, and returns the final result set to the client application.
• Catalog Management: Maintains database metadata, table schemas, view definitions, and security permissions.

Exam Note: The Leader Node does not store user table data and is not billed separately. Data storage and query execution occur strictly on Compute Nodes.

2. Compute Nodes & Node Slices

Compute Nodes execute compiled query code sent by the Leader Node and return intermediate results. Each Compute Node is divided into virtual processing units called Slices.

• Slice Architecture: A slice is assigned a portion of the node's CPU, memory, and disk capacity. The number of slices per node depends on the node instance size (for example, ra3.xlplus has 2 slices, ra3.4xlarge has 4 slices, and ra3.16xlarge has 16 slices).
• Parallel Execution: When data is loaded into Redshift, table rows are distributed across slices according to the table's Distribution Style. During query execution, all slices operate concurrently in parallel.

---

Decoupled Compute & Storage: RA3 Nodes & Redshift Managed Storage (RMS)

Legacy Node Types vs. RA3 Architecture

Historically, Redshift offered coupled compute-and-storage instance types:

• Dense Storage (DS2): Optimized for large data volumes using hard disk drives (HDDs), but limited compute performance.
• Dense Compute (DC2): Optimized for high performance using local NVMe SSDs, but storage was strictly bound to node count.

In coupled architectures, scaling storage required adding compute nodes (increasing costs unnecessarily), while scaling compute forced paying for unneeded storage capacity.

The RA3 Instance Family & RMS

RA3 instances revolutionize Redshift architecture by completely decoupling compute capacity from storage capacity through Redshift Managed Storage (RMS).

How Redshift Managed Storage (RMS) Works

RMS leverages Amazon S3 as the underlying persistent storage tier, offering 99.999999999% (11 9s) of data durability. However, to maintain sub-second query performance, RA3 compute nodes use high-performance local NVMe SSDs as an intelligent, tier-1 data cache.

1. Hot Data Handling: Frequently queried data blocks reside in local NVMe SSD cache.
2. Cold Data Offload: As data grows beyond local SSD capacity, colder data blocks spill over automatically to RMS on S3 without any administrative intervention.
3. High-Bandwidth Networking: RA3 nodes use AWS Nitro System high-bandwidth networking (up to 100 Gbps) to stream data blocks rapidly from RMS into compute slices during heavy scans.

---

Automatic WLM and Short Query Acceleration

Automatic workload management (WLM) assigns memory and adjusts concurrency from observed query resource needs. Heavy joins can run at lower concurrency while lighter scans and inserts run at higher concurrency. Queue priority expresses business importance, and query monitoring rules can log, hop, or cancel queries that cross defined boundaries.

Short query acceleration (SQA) works alongside WLM. Redshift uses machine learning to predict eligible short-running queries and runs them in a dedicated space so they do not wait behind long ETL scans. SQA is enabled by default in the default parameter group and in new parameter groups. It prioritizes short work; it does not make an inefficient large query inherently cheap.

Long ETL scans ---> Automatic WLM queue and memory allocation
Short eligible queries ---> SQA dedicated space ---> Earlier completion

---

Workload Scaling & Elasticity Mechanisms

Redshift provides four major elasticity mechanisms to handle variable analytical workloads:

1. Elastic Resize

Elastic Resize allows you to add or remove nodes from an existing Redshift cluster, or change node types (e.g., from dc2.large to ra3.4xlarge) within minutes. During an Elastic Resize, Redshift pauses active queries, updates slice configurations, re-maps storage pointers in RMS, and resumes database operations. Total downtime is typically under a few minutes.

2. Classic Resize

Classic Resize creates a completely new target cluster in the background, copies data from the source cluster snapshot, and updates DNS endpoints. Classic Resize takes hours or days for multi-terabyte datasets and is primarily used when resizing between incompatible node configurations where Elastic Resize is unsupported.

3. Concurrency Scaling

When hundreds of concurrent business intelligence (BI) users execute queries simultaneously, query queues can form in Redshift Workload Management (WLM). Concurrency Scaling addresses read query spikes by automatically provisioning transient cluster capacity within seconds.

• Automatic Operations: As eligible queues build, Redshift routes supported read or write queries to concurrency scaling capacity automatically.
• Eligibility Matters: Not every query or write pattern is eligible, so monitor queueing and the concurrency-scaling status rather than assuming all ingestion is offloaded.
• Cost Efficiency: Clusters earn 1 hour of free Concurrency Scaling credits for every 24 hours the main cluster runs. Additional usage is billed on a per-second basis.

4. Redshift Serverless

Redshift Serverless removes the requirement to provision or manage data warehouse clusters. You configure base capacity and capacity controls in Redshift Processing Units (RPUs). Redshift Serverless scales capacity for workload demand; compute is metered per second with a 60-second minimum, and managed storage and other applicable usage are billed separately.

---

Code Example: Managing Redshift Provisioned Clusters and Serverless via AWS SDK Boto3

import boto3

# Initialize Redshift & Redshift Serverless clients
redshift_client = boto3.client('redshift', region_name='us-east-1')
serverless_client = boto3.client('redshift-serverless', region_name='us-east-1')

def scale_provisioned_cluster(cluster_identifier: str, target_nodes: int):
    """
    Executes an Elastic Resize on a provisioned RA3 Redshift cluster.
    """
    try:
        response = redshift_client.resize_cluster(
            ClusterIdentifier=cluster_identifier,
            ClusterType='multi-node',
            NodeType='ra3.4xlarge',
            NumberOfNodes=target_nodes,
            Classic=False
        )
        print(f"Elastic Resize initiated for cluster '{cluster_identifier}'. Target nodes: {target_nodes}")
        return response['Cluster']['ClusterStatus']
    except Exception as e:
        print(f"Error initiating Elastic Resize: {str(e)}")
        raise e

def configure_serverless_workgroup(workgroup_name: str, base_rpu: int):
    """
    Updates the base Redshift Processing Unit (RPU) capacity for a Serverless Workgroup.
    """
    try:
        response = serverless_client.update_workgroup(
            workgroupName=workgroup_name,
            baseCapacity=base_rpu
        )
        print(f"Serverless Workgroup '{workgroup_name}' updated with base RPU capacity: {base_rpu}")
        return response['workgroup']['workgroupArn']
    except Exception as e:
        print(f"Error updating Serverless Workgroup: {str(e)}")
        raise e