12.3 Asynchronous Event Processing: SNS, SQS Queues & Dead-Letter Handling

Asynchronous Messaging Patterns: SNS-to-SQS Fan-Out

In distributed cloud systems, coupling microservices synchronously via direct HTTP/REST calls introduces tight operational dependencies, cascading service degradations, and brittle failover mechanics. The Amazon SNS to Amazon SQS Fan-Out pattern provides an asynchronous, loosely coupled architecture where an event published once to an SNS topic is replicated and delivered simultaneously to multiple consumer SQS queues.

Publishing Service (e.g. Order Gateway) 
       │
       ▼
[ Amazon SNS Topic ] (order-events)
       │
       ├──> [ SQS Queue A (Billing Service) ]     ──> Consumer Workers
       ├──> [ SQS Queue B (Inventory Service) ]   ──> Consumer Workers
       ├──> [ SQS Queue C (Fraud Analytics) ]     ──> Consumer Workers
       └──> [ SQS Queue D (Shipping Service) ]    ──> Consumer Workers

Benefits of the Fan-Out Pattern

• Fault Isolation: If the Billing Service crashes or undergoes scheduled maintenance, messages safely accumulate in Queue A without impacting the processing speed of Inventory (Queue B) or Shipping (Queue D).
• Independent Scaling: Each consumer service scales independently based on its own queue depth (ApproximateNumberOfMessagesVisible), using dedicated Auto Scaling policies or AWS Lambda concurrency limits.
• Heterogeneous Consumption: An SNS topic can fan out to SQS queues, AWS Lambda functions, HTTP/HTTPS webhooks, and mobile push notifications simultaneously.

SNS Message Filtering: Message Attributes vs. Message Body

By default, an Amazon SQS queue subscribed to an SNS topic receives every message published to that topic. In large-scale systems, this results in massive waste: worker instances continuously poll and deserialize messages, only to discard 90% of them as irrelevant.

Amazon SNS supports Subscription Filter Policies that evaluate criteria before message delivery. Unmatched messages are dropped immediately at zero cost to the subscriber:

1. Message Attribute Filtering (MessageAttributes): Evaluates key-value metadata attached to the message headers outside the main body.
2. Message Body Filtering (MessageBody): Evaluates JSON properties directly within the message payload, eliminating the need for publishing services to duplicate payload fields into message attributes.

{
  "order_type": ["wholesale", "b2b"],
  "order_total": [{
    "numeric": [">=", 5000]
  }],
  "shipping_address": {
    "country": ["US", "CA"]
  }
}

[!TIP] Cost & Compute Optimization: Subscription filter policies eliminate downstream compute costs entirely. If an SNS topic receives 10,000,000 daily messages and a specialized audit queue requires only 10,000 of them, message filtering prevents 9,990,000 unnecessary SQS messages and millions of wasted Lambda invocations.

Amazon SQS Mechanics: Standard vs. FIFO Queues

Selecting between SQS Standard and SQS FIFO queues is one of the most frequently tested architectural decisions on the DOP-C02 exam.

Comparison Table: Standard vs. FIFO Queues

FIFO Operational Deep Dive

1. MessageGroupId (Concurrency Partitioning)

The MessageGroupId is a mandatory tag for all FIFO messages that specifies the distinct group the message belongs to:

• In-Order Availability: SQS returns messages in order within a MessageGroupId and withholds later messages while earlier received messages remain in flight. If a message is not deleted before its visibility timeout, it can be delivered again, so application processing and state changes must be idempotent.
• Horizontal Parallelism: Messages with different MessageGroupId values can be processed simultaneously by different worker threads or Lambda instances!

[!CAUTION] DOP-C02 Anti-Pattern: One static MessageGroupId serializes the whole workload behind one ordered group. Choose a business key such as customer ID, account ID, device ID, or transaction thread when independent groups can safely run in parallel.

2. MessageDeduplicationId (Producer-Side Send Deduplication)

Every FIFO message requires a deduplication token. If a producer publishes a message with a specific MessageDeduplicationId, any subsequent message published with that same token within the 5-minute deduplication interval is acknowledged as successful by SQS but never delivered to consumers:

• Manual Token: The producer explicitly provides a unique transaction ID or UUID.
• Content-Based Deduplication (ContentBasedDeduplication: true): SQS automatically computes a SHA-256 hash of the entire message body to serve as the deduplication token.

---

Dead-Letter Queues (DLQs) & Failure Handling Mechanics

A Dead-Letter Queue (DLQ) is a secondary SQS queue that isolates messages that cannot be processed successfully by consumer applications. Moving failing messages to a DLQ prevents unparseable or "poison-pill" messages from indefinitely blocking queue processing.

Redrive Policy Configuration and maxReceiveCount Tuning

A redrive policy attached to the source queue defines when messages transition to the DLQ:

{
  "deadLetterTargetArn": "arn:aws:sqs:us-east-1:123456789012:OrderProcessingDLQ.fifo",
  "maxReceiveCount": 3
}

• deadLetterTargetArn: The ARN of the dead-letter queue. The DLQ must match the source queue type (a FIFO queue must target a FIFO DLQ; a Standard queue must target a Standard DLQ).
• maxReceiveCount: The number of times a message can be delivered to consumers before being automatically moved to the DLQ.

Tuning maxReceiveCount Tradeoffs

• Too few receives: Transient errors can move valid messages to the DLQ before the dependency recovers.
• Too many receives: Poison messages consume capacity and, for FIFO, can block a message group for too long.
• Selection rule: Base maxReceiveCount on measured processing time and transient-failure recovery, combine retries with backoff, and alarm on DLQ depth. AWS does not prescribe one universal attempt count.

SNS Subscription Dead-Letter Queues

While SQS DLQs handle consumer processing failures, SNS Subscription DLQs handle delivery failures between SNS and the downstream endpoint (e.g., when SNS cannot deliver to an SQS queue due to an unencrypted vs. KMS-encrypted queue mismatch, missing IAM permissions, or an unreachable HTTPS webhook once the subscription delivery retry policy is exhausted). The DLQ is attached to the subscription itself.

Native SQS DLQ Redrive

Historically, recovering messages from an SQS DLQ required writing custom scripts or Lambda functions to read messages from the DLQ and invoke sqs:SendMessage back into the source queue.

AWS provides native SQS DLQ Redrive via the AWS Management Console or AWS CLI (StartMessageMoveTask):

aws sqs start-message-move-task \
    --source-arn arn:aws:sqs:us-east-1:123456789012:OrderProcessingDLQ.fifo \
    --destination-arn arn:aws:sqs:us-east-1:123456789012:OrderProcessingQueue.fifo

• Redrive Destination: Can redrive messages back to the original source queue or to a custom diagnostic queue.
• Velocity Control: Allows setting a maximum redrive rate (messages per second) to prevent overwhelming newly recovered downstream databases.
• Inspection: Enables inspecting sample messages, headers, and error attributes directly in the console before initiating the move task.

---

AWS Lambda Asynchronous Invocation Dynamics & Destinations

When AWS Lambda is invoked asynchronously (e.g., by Amazon EventBridge, Amazon S3 event notifications, Amazon SNS, or direct SDK calls with InvocationType: Event), Lambda places the event into an internal managed invocation queue.

Asynchronous Retry Lifecycle

Async Invocation ──> [ Internal Lambda Queue ] ──> Worker Execution
                                                     │ (Fails with Exception)
                                                     ▼
                           Wait 1 Minute ──> Attempt 2 (Retry 1)
                                                     │ (Fails with Exception)
                                                     ▼
                           Wait 2 Minutes ─> Attempt 3 (Retry 2)
                                                     │ (Fails: Retries Exhausted)
                                                     ▼
                                       [ Lambda Destination / DLQ ]

• Default Retries: Lambda attempts execution immediately. If the function throws an unhandled error or times out, Lambda automatically retries two additional times (total 3 attempts).
• Retry Delays: Lambda applies exponential backoff between retries (approximately 1 minute after the first failure, and 2 minutes after the second failure).
• Configuration Controls:
  • MaximumRetryAttempts: Configurable from 0 to 2.
  • MaximumEventAgeInSeconds: Configurable from 60 seconds to 21,600 seconds (6 hours). If an event sits in the queue longer than this threshold, it is dropped or sent to a failure destination.

Lambda Dead-Letter Queue vs. Lambda Destinations

Schema of a Lambda OnFailure Destination Payload

{
  "version": "1.0",
  "timestamp": "2026-09-11T16:45:00.000Z",
  "requestContext": {
    "requestId": "a1b2c3d4-e5f6-7a8b-9c0d-1e2f3a4b5c6d",
    "functionArn": "arn:aws:lambda:us-east-1:123456789012:function:OrderHandler:$LATEST",
    "condition": "RetriesExhausted",
    "approximateInvokeCount": 3
  },
  "requestPayload": {
    "orderId": "ORD-99881",
    "amount": 450.00
  },
  "responseContext": {
    "statusCode": 500,
    "executedVersion": "$LATEST"
  },
  "responsePayload": {
    "errorMessage": "Database connection timed out after 5000ms",
    "errorType": "DatabaseConnectionTimeout",
    "stackTrace": [
      "File \"/var/task/index.js\", line 42, in handler"
    ]
  }
}

---

Backoff, Jitter & Retry Engineering

When a downstream dependency (such as Amazon Aurora or an external payment API) suffers a transient outage, hundreds of concurrent Lambda workers or SQS consumers will fail simultaneously. If all consumers retry after exactly 1.0 second, 2.0 seconds, and 4.0 seconds, their retries arrive in synchronized waves, creating a thundering herd (retry storm) that knocks down the recovering dependency.

Full Jitter Algorithm

To prevent synchronized retry storms, distributed architectures employ Exponential Backoff with Full Jitter:

$T_{\text{sleep}} = \text{random}(0, \min(T_{\text{max}}, T_{\text{base}} \times 2^{\text{attempt}}))$

By drawing a random sleep duration uniformly between zero and the calculated exponential backoff ceiling, retries are smoothly distributed across the entire timeline, allowing the recovering database to absorb connection requests gracefully.