7.1 Amazon DynamoDB Partition Keys, Global Secondary Indexes & Data Modeling

7.1 Amazon DynamoDB Partition Keys, Global Secondary Indexes & Data Modeling

DynamoDB Architecture & Physical Partitioning Fundamentals

Amazon DynamoDB is a fully managed, serverless Key-Value and Document NoSQL database engine designed to deliver consistent, single-digit millisecond latency at arbitrary operational scale. Unlike traditional relational database management systems (RDBMS) that run on fixed server instances, DynamoDB distributes data across an abstracted fleet of physical storage nodes managed entirely by AWS.

Physical Partition Allocation & Key Hashing

When a table is created, DynamoDB allocates physical storage nodes called partitions. Data placement across partitions is governed by an internal hashing algorithm:

1. Hash Function Execution: DynamoDB takes the value of the Partition Key (PK) (also called the Hash Attribute) and passes it through an internal hash function whose implementation is managed by DynamoDB.
2. Partition Hash Mapping: The resulting hash output determines the exact physical storage partition where the item will reside.
3. Partition Hard Limits: Each physical partition operates under strict resource limits:
   • Maximum Storage: 10 GB of total data items per partition.
   • Maximum Provisioned Write Capacity: 1,000 Write Capacity Units (WCUs) per partition.
   • Maximum Provisioned Read Capacity: 3,000 Read Capacity Units (RCUs) per partition.

Item (PK: "USER#8842") ---> [ Internal Hash Function ] ---> Hash Value ---> Managed Physical Partition

When a table's total data volume exceeds 10 GB or total provisioned throughput exceeds 1,000 WCUs / 3,000 RCUs, DynamoDB automatically splits physical partitions. Partition splitting divides the key range in half, allocating new physical storage partitions without requiring downtime or application modification.

Hot Partitions & Partition Key Anti-Patterns

Because throughput limits apply at the physical partition level, an unbalanced distribution of access requests leads to a hot partition. If a single partition receives traffic exceeding 1,000 WCUs or 3,000 RCUs, DynamoDB throttles incoming requests and returns an HTTP 400 ProvisionedThroughputExceededException error, even if total table-level capacity is underutilized.

Common Partition Key Anti-Patterns:

• Low Cardinality Keys: Choosing attributes with few distinct values, such as Status (ACTIVE vs INACTIVE) or Gender (M vs F). Millions of items map to the exact same partition.
• Monotonically Increasing Keys: Using timestamps or sequential IDs (2026-08-13-0001, 2026-08-13-0002) as partition keys. All new write operations hit the exact same physical partition handling the current time window.

Remediation Strategies:

• High Cardinality Attributes: Select keys with millions of distinct values, such as UUID, user_id, or device_mac_address.
• Write Sharding (Random Salting): Append a calculated random suffix (0 through N-1) or calculated hash modulo to partition keys (e.g., ORDER#2026-08-13#2). This splits write bursts across multiple partitions. Reads then scatter-gather across all shards.

---

Throughput Capacity Planning: Provisioned vs. On-Demand

DynamoDB offers two billing and capacity management modes:

RCU and WCU Calculation Rules for the Exam

Calculating throughput requirements is a frequent exam requirement. Remember these exact sizing formulas:

Read Capacity Units (RCUs)

• Item Size Basis: Rounded up to the nearest 4 KB increment.
• Eventually Consistent Read (Default): 1 RCU provides 2 reads per second up to 4 KB.
• Strongly Consistent Read: 1 RCU provides 1 read per second up to 4 KB.
• Transactional Read: 2 RCUs provide 1 transactional read per second up to 4 KB.

$\text{RCUs required} = \left( \lceil \frac{\text{Item Size in KB}}{4\text{ KB}} \rceil \right) \times \text{Reads per second} \times \text{Consistency Factor}$ Where Consistency Factor = 1.0 for Strongly Consistent, 0.5 for Eventually Consistent, and 2.0 for Transactional.

Write Capacity Units (WCUs)

• Item Size Basis: Rounded up to the nearest 1 KB increment.
• Standard Write: 1 WCU provides 1 write per second up to 1 KB.
• Transactional Write: 2 WCUs provide 1 transactional write per second up to 1 KB.

$\text{WCUs required} = \left( \lceil \frac{\text{Item Size in KB}}{1\text{ KB}} \rceil \right) \times \text{Writes per second} \times \text{Write Type Factor}$ Where Write Type Factor = 1.0 for Standard Write, and 2.0 for Transactional Write.

---

Single-Table Design & Entity Overloading

In relational databases, normalization divides entities into discrete tables (Customers, Orders, OrderItems) linked by foreign keys. Querying normalized data requires high-latency table JOIN operations. DynamoDB does not support JOIN operations.

In DynamoDB NoSQL architecture, Single-Table Design consolidates multiple relational entities into a single DynamoDB table. Access patterns dictate the table schema, rather than entity relationships.

Overloading Primary Keys (PK & SK)

Single-table schemas use generic attribute names like PK (Partition Key) and SK (Sort Key) to store disparate record types:

Item Collections

An Item Collection consists of all items in a table (or index) that share the exact same Partition Key (PK) value across different Sort Keys (SK).

Using a single Query API call with key condition expressions (such as PK = :cust_id AND SK BEGINS_WITH("ORDER#")), an application can fetch matching order items in a single HTTP network round-trip; omit the sort-key condition when the result must also include the profile item with single-digit millisecond latency.

---

Secondary Indexes: Global Secondary Indexes (GSIs) vs. Local Secondary Indexes (LSIs)

When access patterns require querying attributes other than the base table's primary key, DynamoDB secondary indexes create alternative projection views of table data.

Critical Exam Warning: GSI Write Backpressure

Because DynamoDB asynchronously replicates base table writes to GSIs, if a GSI runs out of provisioned Write Capacity Units (WCUs), the GSI rejects updates. To preserve data consistency between base table and index, DynamoDB will throttle writes on the base table itself, throwing a ProvisionedThroughputExceededException on base table write operations! Always ensure GSI write capacity matches or exceeds base table write velocity.

Index Projections

When creating secondary indexes, you choose which attributes to project into the index:

• KEYS_ONLY: Only base table PK, SK, and index keys are projected. Lowest storage cost.
• INCLUDE: Keys plus specifically selected non-key attributes are projected.
• ALL: Every attribute from the base table is projected into the index. Highest storage cost, but avoids base table fetches.

---

Advanced Data Engineering Features: DAX, Streams & Global Tables

DynamoDB Accelerator (DAX)

DAX is a fully managed, highly available in-memory write-through cache designed specifically for DynamoDB. It reduces read latencies from single-digit milliseconds to microseconds (< 1 ms).

• Architecture: Runs inside a Private Subnet in your VPC. Client SDKs replace standard DynamoDB endpoints with DAX cluster endpoints.
• Cache Mechanisms: Item cache (caches individual GetItem calls) and Query cache (caches Query and Scan result sets).

DynamoDB Streams & Change Data Capture (CDC)

DynamoDB Streams captures an item-level, time-ordered sequence of write operations (INSERT, MODIFY, REMOVE) on a table, retaining records for 24 hours.

Stream View Types:

• KEYS_ONLY: Only key attributes of modified items.
• NEW_IMAGE: Entire item as it appears after modification.
• OLD_IMAGE: Entire item as it appeared before modification.
• NEW_AND_OLD_IMAGES: Both new and old item representations.

Common stream consumption pattern: DynamoDB Stream trigger $\rightarrow$ AWS Lambda $\rightarrow$ Amazon OpenSearch Service for full-text search indexing or S3 Data Lake landing.

DynamoDB Global Tables

Global Tables provides multi-region, multi-active replication for globally distributed applications. Leveraging underlying DynamoDB Streams, writes to any regional table replica are asynchronously replicated across selected AWS regions with typical latency under one second.

---

Time to Live for Lifecycle Expiration

DynamoDB Time to Live (TTL) expires items from a table using one designated attribute containing a Unix epoch timestamp in seconds as a Number. An expired timestamp makes an item eligible for background deletion; deletion is not immediate and AWS generally completes it within a few days. Applications should filter expired items if they must stop returning them at the deadline.

TTL deletion does not consume write capacity in the Region where the expiration occurs. It appears in DynamoDB Streams as a service deletion, which can trigger cleanup of derived data. With Global Tables, the initial expiration does not consume local write units, but each replicated delete consumes replicated write capacity in replica Regions. TTL is appropriate for sessions, idempotency records, or other retention windows; use an explicit conditional/application delete when the business requires exact-time removal.

Code Example: Python Boto3 Single-Table Querying and GSI Creation

import boto3
from boto3.dynamodb.conditions import Key

# Initialize DynamoDB resource
dynamodb = boto3.resource('dynamodb', region_name='us-east-1')
table = dynamodb.Table('ECommerceSingleTable')

def query_customer_orders(customer_id: str, start_date: str):
    """
    Queries single-table schema for a specific customer's orders 
    placed after start_date using KeyConditionExpression.
    """
    try:
        response = table.query(
            KeyConditionExpression=Key('PK').eq(f"CUST#{customer_id}") & 
                                  Key('SK').begins_with(f"ORDER#{start_date}"),
            ConsistentRead=True  # Strongly consistent read on base table
        )
        items = response.get('Items', [])
        print(f"Retrieved {len(items)} order records for customer {customer_id}")
        return items
    except Exception as e:
        print(f"Error querying DynamoDB table: {str(e)}")
        raise e

def query_gsi_by_order_status(status: str):
    """
    Queries Global Secondary Index (GSI1) to fetch all orders by status across customers.
    """
    try:
        response = table.query(
            IndexName='GSI1-Status-Date-Index',
            KeyConditionExpression=Key('GSI1_PK').eq(f"STATUS#{status}"),
            ConsistentRead=False # GSIs only support eventually consistent reads
        )
        return response.get('Items', [])
    except Exception as e:
        print(f"GSI Query Failed: {str(e)}")
        raise e