7.2 Advanced SQS Patterns — DLQ, Backpressure, and FIFO
Key Takeaways
- Dead-Letter Queues (DLQs) capture messages that fail processing after maxReceiveCount attempts, preventing poison pills from blocking the main queue.
- SQS-based backpressure uses queue depth to control the rate at which work is consumed, preventing downstream services from being overwhelmed.
- SQS FIFO queues support message group IDs for ordered processing within groups and deduplication IDs for exactly-once processing.
- Long polling (WaitTimeSeconds up to 20 seconds) reduces SQS API costs by eliminating empty response loops.
- SQS temporary queues implement the request-response pattern for synchronous-style communication over asynchronous infrastructure.
Advanced SQS Patterns — DLQ, Backpressure, and FIFO
Quick Answer: DLQ = catch failed messages after N retries. FIFO = ordered + exactly-once (3000 msg/s with batching). Long polling = reduce API costs (wait up to 20s for messages). Backpressure = use queue depth to control downstream load. Message Group ID = parallel FIFO processing within groups.
Dead-Letter Queue (DLQ) Patterns
How DLQ Works
- Message arrives in main queue
- Consumer receives and processes the message
- If processing fails, message returns to queue (after visibility timeout)
- After maxReceiveCount failures, message moves to DLQ
- DLQ messages can be investigated, fixed, and redriven
DLQ Configuration
| Setting | Description |
|---|---|
| maxReceiveCount | Number of receive attempts before moving to DLQ (e.g., 3) |
| DLQ retention | Up to 14 days (set higher than main queue) |
| Redrive | Move messages from DLQ back to source queue after fixing |
| Alarm | Set CloudWatch alarm on DLQ depth to alert on failures |
DLQ Best Practices
- Set DLQ retention longer than the source queue
- Set up CloudWatch alarms on DLQ message count
- Use redrive policy to replay messages after fixing the issue
- DLQ for a Standard queue must be Standard; DLQ for a FIFO queue must be FIFO
FIFO Queue Advanced Features
Message Group ID
Messages with the same Message Group ID are processed in strict order. Messages with different Group IDs can be processed in parallel.
| Scenario | Group ID Strategy | Behavior |
|---|---|---|
| All messages must be in order | Same Group ID for all | Serial processing |
| Orders from different customers | Customer ID as Group ID | Parallel per customer, serial per customer |
| Events from different devices | Device ID as Group ID | Parallel per device |
Deduplication
| Method | How |
|---|---|
| Content-based | SQS generates hash of message body; duplicates within 5 minutes are rejected |
| Deduplication ID | You provide an explicit ID; duplicates within 5 minutes are rejected |
FIFO Throughput
| Mode | Throughput |
|---|---|
| Standard FIFO | 300 messages/second (send, receive, delete per API) |
| High Throughput FIFO | 3,000 messages/second with batching, 300 without |
Backpressure Pattern
Use SQS queue depth to control the rate of processing:
- Producers send messages to SQS at any rate
- SQS absorbs the burst (buffers messages)
- Consumers process at their own pace
- Auto Scaling adds/removes consumers based on ApproximateNumberOfMessagesVisible metric
- Downstream services are never overwhelmed
This pattern decouples production rate from consumption rate.
Long Polling
| Polling Type | Behavior | API Calls | Cost |
|---|---|---|---|
| Short polling | Returns immediately (even if empty) | High (polling in tight loop) | Higher |
| Long polling | Waits up to 20 seconds for messages | Lower | Lower |
Enable long polling by setting WaitTimeSeconds > 0 (queue level or per-request).
On the Exam: "Reduce SQS costs" → Enable long polling (WaitTimeSeconds=20). "Handle messages that repeatedly fail" → Configure DLQ with maxReceiveCount. "Process messages in order per customer" → FIFO queue with Customer ID as Message Group ID.
A company processes financial transactions that must be processed exactly once and in the exact order they were submitted per account. How should they configure SQS?
What is the purpose of setting WaitTimeSeconds to 20 on an SQS queue?