Memory hook: Queue work for competing workers, fan out events to independent consumers, and retain streams when replay matters.
Must remember
SQS separates arrival rate from processing rate
- Standard queues provide at-least-once delivery and best-effort ordering. More workers can process different messages concurrently; producer and consumer availability no longer have to match exactly.
- Retention determines how long an unprocessed message can remain. Visibility timeout temporarily hides a received message while a worker processes it. Receiving does not delete it; delete only after successful processing.
- Set visibility around realistic processing/retry behavior. A worker can extend visibility for long work. If it crashes or visibility expires before deletion, another attempt can occur.
- Long polling waits for available messages, reducing empty receives and their cost. It does not extend retention or the worker's processing deadline.
- A DLQ receives messages after the configured receive-count failure threshold. Investigate and correct the cause before redriving; a DLQ is not successful completion.
- Queue resource policies authorize cross-service or cross-account senders, often with source ARN/account conditions. Encryption and permission to send are separate controls. Visibility behavior
FIFO ordering is scoped, and business effects still need protection
- FIFO queues preserve order within a message group. Different groups can progress independently; one global group limits parallelism.
- Deduplication IDs prevent duplicate sends within the five-minute deduplication interval. Content-based deduplication hashes the message body, not its attributes; repeated identical bodies need deliberate identity semantics.
- Deduplicated enqueueing is not a transaction with a payment provider or database. A worker may commit a side effect, crash before acknowledging, then receive the message again. Use an idempotency key and atomic/conditional business-state handling.
- Moving a message out to a DLQ can interrupt an application's intended sequence. If later messages must never overtake a failed operation, design the failure workflow accordingly. FIFO deduplication
SNS distributes copies; it does not create a worker backlog by itself
- SNS pushes messages to subscribers. Filter policies select messages by configured attributes or payload fields; they are not authorization policies.
- SNS plus SQS fan-out gives each subscriber its own copy, buffer, scaling and retry boundary. A single shared queue instead distributes work among competing consumers.
- SNS FIFO topics with compatible FIFO queue subscriptions support ordered fan-out; do not assume all subscriber types preserve the same ordering guarantees.
- S3 events can publish through SNS for independent consumers. Direct S3-to-SQS-FIFO notification is unsupported; EventBridge is a supported routing alternative. See object events. Messaging choices
Streams and broker compatibility answer different requirements
- Kinesis Data Streams retains records so independent consumers can read and replay them. A partition key maps records to a shard; ordering is scoped to the relevant shard/key, not the entire distributed workload.
- Provisioned mode means planning shard capacity; on-demand mode reduces that capacity-management work. Neither excuses a design that sends all traffic through one hot partition key.
- Consumers track processing position/checkpoints. Reading a record does not delete it for other applications. Retention and consumer lag determine whether replay remains possible. Kinesis Data Streams
- Amazon Data Firehose provides managed delivery into supported destinations, with buffering and optional transformation/format conversion. Choose it when delivery is the goal; choose a stream plus consumers when custom processing/replay is the goal. Firehose's buffering is not a general-purpose long-term replay contract. Firehose
- Amazon MQ manages supported ActiveMQ/RabbitMQ brokers. Existing JMS/AMQP-style applications that must preserve broker semantics may fit MQ better than rewriting around SQS/SNS.
- ASG worker scaling: approximate acceptable backlog per instance as acceptable queueing delay divided by average processing time. Scale on backlog per worker, not raw queue length alone; keep retries and downstream limits in the design.
Choose under exam pressure
| Requirement in the question | Best direction |
|---|---|
| Buffer bursts before independent jobs | SQS |
| Per-order sequencing with parallel unrelated orders | FIFO with a group per order/workflow |
| Every application must receive the event | SNS fan-out with separate queues |
| Replay telemetry through several consumers | Kinesis Data Streams |
| Managed streaming delivery into S3 | Firehose |
| Preserve an existing broker protocol | Amazon MQ |
| Scale workers while meeting queueing latency | Backlog per worker |
Traps
- Visibility, retention and delay are different clocks.
- FIFO does not make an external payment automatically exactly once.
- A filter decides delivery selection; a resource policy decides whether delivery is authorized.
- A partition key's ordering benefit can become a throughput bottleneck when too much work shares that key.
Active recall
1. A worker takes 70 seconds but visibility is 30 seconds. Why are side effects duplicated?
Another attempt can start before the first worker finishes. Increase or extend visibility appropriately, and use idempotency because failures can still happen after a side effect but before acknowledgement.
2. Audit and image-processing applications both need every upload event. Why not give them one queue?
They would compete for messages. Give each its own subscriber queue so each receives a copy and has independent retries, backlog and capacity.
3. A FIFO producer sends identical bodies with different attributes. Can content-based deduplication distinguish them?
Not solely by those attributes: the hash uses the body. Use deliberate deduplication IDs or distinct message content when these are genuinely separate business events.
4. A new analytics consumer must replay yesterday's retained telemetry. Why is a successfully drained work queue insufficient?
Completed queue messages have been removed. A retained stream with appropriate retention and consumer checkpoints is designed for independent replay.
5. A team only needs buffered delivery into S3, while another must keep its RabbitMQ protocol. Should both start with Kinesis?
No. Firehose matches managed delivery; Amazon MQ matches broker compatibility. Kinesis is appropriate when a retained stream and custom independent consumers are part of the requirement.
Terraform anchor: Encode queue policies, redrive rules and subscription filters as structured HCL with jsonencode; the provider expects JSON strings at those boundaries.