SQS queues and the dead-letter queue

In 1 minute

An SQS queue (Simple Queue Service) keeps messages until deletion or their retention limit. When a consumer receives a message, the queue hides it for the visibility timeout; if it is not deleted in time, it shows up again. A dead-letter queue (DLQ) is a normal queue that receives messages the main queue gave up on. MiniTickets moves a message after 3 receives without deletion; an alarm watches for any visible message in the jobs DLQ.

Picture it

flowchart TB
  T["SNS topic"] --> Q["jobs queue<br/>visibility 60 s"]
  Q -->|"batch"| W["worker Lambda"]
  W -->|"failed: hide 10 s, 40 s, 120 s"| Q
  Q -->|"4th receive"| D["jobs DLQ<br/>retention 14 days"]
  D -->|"1 message"| A["alarm → email"]

How it works

1. A message arrives from the SNS topic and waits (up to 4 days).
2. The consumer receives it. The queue hides it for 60 s, six times the worker’s 10 s timeout.
3. Success: the consumer deletes it. Failure: the worker hides it again, for longer each time (10 s, 40 s, 120 s), so a struggling service gets time to recover.
4. Redrive policy. maxReceiveCount is 3: on the 4th receive SQS moves the message to the DLQ instead of handing it out.
5. A person looks. Nothing reads the DLQ automatically. Someone finds the cause, fixes it, then moves the messages back (“Start DLQ redrive”).

In MiniTickets

Queue	Visibility	Total retention	DLQ after
minitickets-dev-jobs (email)	60 s	4 days	3 receives → minitickets-dev-jobs-dlq
minitickets-dev-tickets	180 s (6 × 30 s)	4 days	3 receives → minitickets-dev-tickets-dlq
Both DLQs	–	14 days	–

Both come from infra/src/jobs/queues.ts → createJobQueues. Encryption at rest is SSE-SQS (a key SQS manages, free). Each DLQ’s redrive allow policy (byQueue) accepts messages from its own source queue only.

SES refuses an email. We follow the message through its retries into the dead-letter queue, until a person gets an alarm email.

1. Jobs SQS queue → Worker Lambda. The worker gets the message in a batch.
2. Worker Lambda → SES email identities. SES answers MessageRejected: Email address is not verified (the account is in the sandbox).If it breaks: Or the DKIM records are missing, and every send fails. Look: SES → Identities → minitickets.theversehub.in → Authentication (DKIM).
3. Worker Lambda → Jobs SQS queue. The handler lists the message in batchItemFailures and hides it 10 s, then 40 s, then 120 s (ChangeMessageVisibility). A normal return retries only listed failures; a timeout or uncaught exception retries the whole batch.If it breaks: Without ReportBatchItemFailures, Lambda ignores our returned failure list and deletes failed messages too after a normal return. Look: aws lambda list-event-source-mappings: FunctionResponseTypes.
4. Jobs SQS queue → Jobs dead-letter queue. On the 4th receive (maxReceiveCount 3), SQS moves the message to minitickets-dev-jobs-dlq.
5. Jobs dead-letter queue → CloudWatch alarms. jobs-dlq-not-empty checks whether ApproximateNumberOfMessagesVisible exceeds 0 for one 60 s period. Approximate metrics and alarm evaluation can delay detection.
6. CloudWatch alarms → Alerts SNS topic. The alarm publishes to minitickets-dev-alerts, which emails alertEmail.If it breaks: The email subscription was never confirmed: nobody hears it. Look: SNS → minitickets-dev-alerts → Subscriptions.
7. Saved Logs Insights queries → CloudWatch log groups. The person runs minitickets-dev/errors, follows the correlationId, fixes the cause, then redrives the DLQ.

Play the failing email on the map →

See it in AWS

Console

SQS → Queues → minitickets-dev-jobs-dlq → Send and receive messages → Poll for messages. After the fix: Start DLQ redrive.

CLI

aws sqs get-queue-attributes --queue-url <jobs-dlq-url> --attribute-names ApproximateNumberOfMessages

Code

The waits: apps/worker/src/backoff.ts → backoffSeconds.

Break it

Symptom	Cause	Where to look
jobs-oldest-message-age fires	Nothing reads the queue: the worker is off, failing, throttled, or its trigger is disabled	Lambda → Triggers
The same email twice within a minute	The visibility timeout is shorter than the handler’s run time	Queue → visibility timeout
The DLQ fills with one message per booking	Unverified SES sandbox recipients: “Email address is not verified”	minitickets-dev/errors query

Say it in an interview

Say it like this (20-30 seconds)

Email jobs wait in an SQS queue. A received message is hidden for 60 seconds, six times the Lambda timeout. On failure the worker hides it for 10, 40, then 120 seconds. With maxReceiveCount 3, the fourth receive moves it to a dead-letter queue with 14-day total retention. An alarm watches for visible messages there; a person fixes the cause and redrives. Delivery is at least once, so the worker must handle duplicates.

Follow-ups

Why must the DLQ keep messages longer than the main queue?

In a standard queue a message’s age counts from when it was first sent. A DLQ with a shorter retention could delete a message the moment it arrives.

Why six times the Lambda timeout?

Lambda may retry a throttled call with the same batch. A shorter visibility timeout lets the message reappear while it is still being handled, and the email goes out twice.

What is the trade-off of maxReceiveCount 3?

A real outage longer than about three minutes sends healthy messages to the DLQ, so they need a redrive. A higher count waits longer before a person hears about a broken message.

Why not a FIFO queue?

We do not need ordering. FIFO adds message-group choices and still cannot make an email send atomic with its completion record. Our completed-job check skips recorded jobs, but crashes after sending or concurrent copies can still send an email twice.

Advanced: Details: settings

Long polling waits up to 20 s (receiveWaitTimeSeconds). The redrive allow policy is its own resource, because the queue names the DLQ and the DLQ names the queue. Locally the same code runs against ElasticMQ; pnpm jobs:dlq lists and pnpm jobs:redrive moves DLQ messages back. AWS recommends six times the Lambda timeout and explains standard-queue DLQ retention.

Checkpoint

Answer these without scrolling up. If one feels hard, reread that section.

1. What does the visibility timeout do?
2. How many receives before a message goes to the DLQ?
3. Why is the jobs queue’s visibility timeout 60 s and the tickets queue’s 180 s?
4. What does the redrive allow policy prevent?
5. Who reads the DLQ, and what do they do after the fix?

Redraw it from memory: SNS → jobs queue (60 s) → worker → on failure hide 10/40/120 s → 4th receive → DLQ → alarm → email

Try this: on the map, play the failing-email flow and open 'What if it breaks?' at the retry step

Mark this page as read