Decoupled Messaging with Amazon SQS and Amazon SNS

Why queues and topics implement Task 2.4

SAP-C02 Task 2.4 lists application integration (Amazon SNS, Amazon SQS, AWS Step Functions) and the skill implementing loosely coupled dependencies. A HarborCart checkout API that calls inventory, payment, loyalty, and email in one synchronous chain fails when any dependency is slow. The Professional pattern is: accept the order, persist an intent, and let downstream workers proceed independently. Amazon Simple Queue Service (Amazon SQS) is the durable buffer. Amazon Simple Notification Service (Amazon SNS) is the fan-out notifier. Amazon EventBridge is the event bus you pick when routing is content-based or the producer is a SaaS partner—this section teaches that decision without turning SQS into EventBridge.

Amazon SQS standard versus FIFO

Standard queues are the default. AWS documents a nearly unlimited number of SendMessage, ReceiveMessage, and DeleteMessage API calls per second. Delivery is at-least-once: the highly distributed architecture can deliver a duplicate, and messages may arrive out of order, though SQS makes a best-effort attempt to preserve send order. SQS redundantly stores a message in multiple Availability Zones before acknowledging SendMessage.

FIFO queues names must end with .fifo. Ordering and deduplication are per message group (MessageGroupId). HarborCart sets MessageGroupId to orderId so order 111 stays ordered even while order 222 processes in parallel. Different group IDs interleave. Exactly-once processing means SQS FIFO removes duplicates that arrive within the 5-minute deduplication interval when you supply a MessageDeduplicationId or enable content-based deduplication (SHA-256 of the body). Consumers must still be idempotent; FIFO is not a license to skip upserts.

Throughput is the FIFO tax. Without high-throughput mode, each FIFO partition supports 300 transactions per second per API action, or 3,000 messages per second with batching (300 API calls times 10 messages). High throughput for FIFO sets deduplication scope to message group and throughput limit to per message group ID. AWS then publishes regional high-throughput quotas (for example, up to 70,000 TPS and 700,000 batched messages per second in Amazon SQS message quotas for US East (N. Virginia), US West (Oregon), and Europe (Ireland), with lower defaults in other Regions). Spread unique MessageGroupId values or a single hot group becomes the bottleneck. A single group ID for the entire site serializes the queue.

Maximum message size is 1,048,576 bytes (1 MiB). Larger payloads use the extended client libraries and a pointer to Amazon S3 (payload up to 2 GB). Default retention is 4 days (minimum 60 seconds, maximum 14 days). Delay queues and per-message timers go up to 15 minutes. Long polling waits up to 20 seconds and should be the default to cut empty-receive cost.

Visibility timeout and dead-letter queues

Visibility timeout hides a received message from other consumers while you work. The default is 30 seconds; the minimum is 0; the maximum is 12 hours. If processing exceeds the timeout, the message becomes visible and another worker may process it—the usual source of "duplicate charges" in order flows. Extend with ChangeMessageVisibility for long jobs, or set visibility to 0 to nack immediately. There is still no absolute guarantee a standard-queue message will not be delivered twice during the timeout; idempotency remains mandatory.

When AWS Lambda is the consumer, AWS recommends setting the queue visibility timeout to at least six times the function timeout, plus MaximumBatchingWindowInSeconds. Lambda hides the batch for the queue's visibility timeout. If the function errors, the whole batch becomes visible again unless you use partial batch failure reporting. Consumers must tolerate duplicates.

A dead-letter queue (DLQ) receives messages after maxReceiveCount receives. Pair every production queue with a DLQ, alarm on ApproximateNumberOfMessagesVisible on the DLQ, and use redrive after you fix the poison payload. The DLQ must be the same type (FIFO DLQ for a FIFO source). SNS subscriptions and EventBridge targets also support DLQs; a failed push is not the same as an SQS retry loop, but the operational habit is the same: never let a poison message retry forever without an operator signal.

HarborCart's order worker times out at 25 seconds. Visibility timeout of 30 seconds races the worker. They set visibility to 3 minutes, maxReceiveCount to 5, and a FIFO DLQ named orders-dlq.fifo. Payment capture is idempotent on orderId plus an idempotency key stored in DynamoDB.

Amazon SNS fan-out and filter policies

Amazon SNS is publish/subscribe. One Publish to a topic pushes to every matching subscription: SQS queues, Lambda (standard topics), HTTP/HTTPS, email, SMS, mobile push, and Amazon Data Firehose. SNS does not keep a long-lived mailbox the way SQS does. If the subscriber is down, SNS retries according to the protocol; it does not wait 14 days. The reliable pattern for HarborCart is SNS to SQS: each consumer owns a queue, so a warehouse outage does not drop notifications that loyalty already processed.

Subscription filter policies evaluate message attributes (default) or the message body (FilterPolicyScope = MessageBody) before delivery. HarborCart publishes order-confirmed with attribute eventType. Warehouse SQS filters eventType = warehouse. Loyalty Lambda filters eventType = loyalty. Without filters, every subscriber receives every message and filters in code—waste and a blast-radius bug.

SNS FIFO topics (name ends with .fifo) provide ordering and deduplication similar to SQS FIFO. AWS documents delivery to both SQS FIFO and SQS standard queues. Use SNS FIFO plus SQS FIFO when the consumer must keep order. Subscribing a standard queue is allowed when that consumer can tolerate best-effort order and you want to share the topic. Lambda is not a direct FIFO-topic subscriber; subscribe an SQS queue, then attach Lambda to the queue.

SNS publishes cap at 256 KB. SQS now accepts 1 MiB. If the notification is large, use a claim-check: store the body in S3 and publish the URL. Do not invent a larger SNS limit.

EventBridge versus SNS

Both can fan out. They are not interchangeable on SAP-C02.

Pick SNS when HarborCart must notify humans and systems from one order-confirmed publish: email the shopper, drop a message on the warehouse SQS queue, invoke loyalty, and optionally Firehose to S3. Pick EventBridge when the producer is "an EC2 state change," "a partner SaaS webhook already integrated," or "we must replay yesterday's events into a new consumer." You can subscribe SQS to EventBridge, or subscribe SQS to SNS; chaining SNS into EventBridge is usually extra hops without a requirement.

HarborCart order processing

Checkout returns 202 after SendMessage to orders.fifo with MessageGroupId = orderId. A Lambda (or ECS) worker receives the message, captures payment, writes order state, then Publish to a standard SNS topic order-lifecycle with attributes. Filter policies feed:

• Warehouse SQS standard queue (can reorder picks across orders).
• Loyalty Lambda.
• Email protocol for shopper confirmation.
• Firehose for an analytics lake.

Payment capture stays on the FIFO queue so two workers never apply capture then refund out of order for the same orderId. Notifications fan out on SNS because warehouse, loyalty, and email must not share one competing-consumer queue—the classic exam miss is "one SQS queue, three different worker types," which is competing consumers stealing work, not fan-out.

Traps

1. Standard SQS where the stem requires per-order ordering or deduplication.
2. One MessageGroupId for all orders, serializing FIFO.
3. Visibility timeout shorter than processing, causing duplicate side effects.
4. No DLQ, so poison messages cycle forever.
5. Three specialized consumers on one queue instead of SNS (or EventBridge) fan-out.
6. Direct Lambda on an SNS FIFO topic.
7. SNS when the stem needs archive/replay or a SaaS partner bus.
8. Treating API keys, IAM, and SNS filters as the same control.