1.3 Event-Driven Streaming Ingestion with DynamoDB Streams & Lambda

1.3 Event-Driven Streaming Ingestion with DynamoDB Streams & Lambda

Change Data Capture (CDC) Architecture

In event-driven cloud architectures, capturing database modifications as a real-time event stream is known as Change Data Capture (CDC). CDC enables downstream decoupled architectures—such as invalidating caches, updating search indexes in OpenSearch, updating data warehouse tables, or triggering serverless workflows—without performing expensive periodic database polling.

AWS provides native CDC capabilities for Amazon DynamoDB through two streaming integrations: DynamoDB Streams and Amazon Kinesis Data Streams for DynamoDB.

---

DynamoDB Streams Fundamentals

DynamoDB Streams captures a time-ordered sequence of item-level modifications (INSERT, MODIFY, REMOVE) occurring within a DynamoDB table. The log entries are written asynchronously to the stream within milliseconds of the database update.

Key Characteristics of DynamoDB Streams

• Ordering & Scope: Record modification events are strictly ordered per item primary key. Events for distinct items may arrive in arbitrary order.
• Shard Lifecycle: Stream records are organized into shards, which represent a log of table modifications. Shards auto-split and merge in tandem with underlying DynamoDB physical partition scaling.
• 24-Hour Fixed Retention: Stream records are retained for exactly 24 hours. Data older than 24 hours is automatically trimmed and permanently lost. Retention period is non-configurable.
• Zero Impact on Read Capacity: Polling or reading records from DynamoDB Streams does NOT consume table Read Capacity Units (RCUs).

Stream View Types

When enabling DynamoDB Streams, you must specify the Stream View Type, which defines what information is captured in each stream record:

---

Kinesis Data Streams Integration for DynamoDB

For advanced analytics pipelines, AWS allows streaming DynamoDB table changes directly into Amazon Kinesis Data Streams instead of standard DynamoDB Streams.

Architectural Comparison: DynamoDB Streams vs. Kinesis Data Streams

Decision Guidance: Choose native DynamoDB Streams for lightweight, low-cost Lambda triggers requiring <24h retention. Choose Kinesis Data Streams for DynamoDB when requiring retention >24h, fan-out to >2 consumers, or direct ingestion into Amazon S3 via Firehose.

---

AWS Lambda Event Source Mapping (ESM)

AWS Lambda integrates with streaming sources (DynamoDB Streams, Kinesis, MSK) via an internal infrastructure component called the Event Source Mapping (ESM). The ESM runs a polling loop on behalf of your serverless architecture, fetching batches of records from stream shards and invoking your Lambda function synchronously.

Tuning ESM Batching & Concurrency

1. Batch Size (BatchSize): Configures the maximum number of records (1 to 10,000) passed to a single Lambda invocation.
2. Maximum Batching Window (MaximumBatchingWindowInSeconds): Sets the maximum time (0 to 300 seconds) Lambda waits to gather records before invoking the function. If BatchSize is not met, Lambda invokes the function as soon as the window expires.
3. Parallelization Factor (ParallelizationFactor): By default, Lambda processes each stream shard with 1 concurrent invocation to guarantee ordered processing. Setting ParallelizationFactor (from 1 to 10) divides a single shard into concurrent sub-streams grouped by partition key, allowing up to 10 concurrent Lambda invocations per shard while preserving strict partition key ordering!

Resilient Error Handling & Dead Letter Destinations

When a Lambda function throws an unhandled exception while processing a stream batch, the ESM halts shard processing and retries the entire batch repeatedly until records expire (24 hours). This causes a blocked shard (poison pill block).

To prevent shard blocking, configure advanced ESM error controls:

• BisectBatchOnFunctionError: When set to True, if a batch invocation fails, Lambda automatically splits the failing record batch into two smaller sub-batches and retries them separately. It recursively bisects until the exact failing record is isolated.
• MaximumRetryAttempts & MaximumRecordAgeInSeconds: Limits how many times Lambda retries a failing record batch or drops records older than a set threshold.
• On-Failure Destination (DestinationOnFailure): Automatically routes metadata about dropped or failed records directly to an Amazon SQS queue or Amazon SNS topic for offline inspection and replay.

---

Idempotent Stream Processing Patterns

Because stream processing guarantees at-least-once delivery, network retries or Lambda function re-invocations can deliver the same stream event multiple times. Lambda functions MUST be designed to be idempotent (processing the same event multiple times produces the exact same side-effect).

Idempotency Techniques

1. Deterministic Unique Keys: Use the DynamoDB Streams eventID or business transaction ID as an idempotency key.
2. DynamoDB Conditional Writes: Maintain a dedicated tracking table in DynamoDB. Before processing, insert the eventID using a conditional expression: attribute_not_exists(event_id) If the insert succeeds, process the payload. If a ConditionalCheckFailedException is raised, skip processing as it is a duplicate event.

---

Code Example: Idempotent Python Lambda Handler for DynamoDB CDC

import json
import os
import time
import boto3
from botocore.exceptions import ClientError

dynamodb = boto3.resource('dynamodb')
PROCESSED_EVENTS_TABLE = os.environ.get('PROCESSED_EVENTS_TABLE', 'ProcessedStreamEvents')
table = dynamodb.Table(PROCESSED_EVENTS_TABLE)

def lambda_handler(event, context):
    records = event.get('Records', [])
    print(f"Processing batch of {len(records)} DynamoDB stream records.")
    
    for record in records:
        event_id = record['eventID']
        event_name = record['eventName']  # INSERT, MODIFY, REMOVE
        
        # Attempt idempotent registration
        try:
            table.put_item(
                Item={'event_id': event_id, 'ttl': int(time.time()) + 86400},
                ConditionExpression='attribute_not_exists(event_id)'
            )
        except ClientError as e:
            if e.response['Error']['Code'] == 'ConditionalCheckFailedException':
                print(f"Duplicate event detected: {event_id}. Skipping execution.")
                continue
            raise e
            
        # Parse CDC Image Payload
        ddb_data = record.get('dynamodb', {})
        new_image = ddb_data.get('NewImage', {})
        old_image = ddb_data.get('OldImage', {})
        
        print(f"Executing CDC event {event_name} for ID {event_id}")
        # Perform downstream integration logic here (e.g. forward to S3/OpenSearch)
        
    return {"statusCode": 200, "body": json.dumps("Stream processing batch completed.")}