The dead-letter queue keeps growing

Symptom

Troubleshooting scenario: minitickets-dev-jobs-dlq-not-empty fires. The worker logs “message failed” and SES reports “Email address is not verified”. A dead-letter queue (DLQ) holds messages the source queue stopped retrying.

Read the error first

SQS moved the message after the receive limit; SES explains this scenario’s failure. Verification is an SES sending rule, not a password or IAM grant. Under Reach / Allowed / Password, classify it as service acceptance. For other messages, distinguish an IAM missing allow from an explicit deny using their actual error. A disabled worker alone leaves a backlog; it does not consume receives.

Decide

flowchart TB
  A["DLQ has messages"] --> B{"Worker error for same job?"}
  B -->|"verification"| C["Check SES sandbox and identities"]
  B -->|"invalid message"| D["Check schema and raw SNS delivery"]
  B -->|"no handler log"| E["Check throttles and trigger"]
  C --> F["Repair cause before controlled redrive"]
  D --> F
  E --> F

Likely causes

Ranked for this DLQ scenario. Only the first explains the quoted SES error; match each job.

1. An unverified recipient in the SES sandbox, or an unverified sending identity in this Region.
2. Invalid JSON/schema, including an SNS envelope when raw delivery is off.
3. Repeated dependency failures, invocation timeouts or throttling exhaust receives.

How to check

Console

SQS → Queues → minitickets-dev-jobs-dlq → Monitoring. Find the matching job in CloudWatch → Logs → Log groups → /minitickets/dev/worker; inspect SES → Account dashboard and Configuration → Identities in the worker’s Region. Inspect message bodies only privately: they contain personal data. Tickets use their own DLQ and log group.

CLI

Set JOBS_DLQ_URL to its URL from SQS; this reads counts without receiving messages.

export AWS_PROFILE=ticketlite
aws sqs get-queue-attributes --queue-url "$JOBS_DLQ_URL" --attribute-names ApproximateNumberOfMessages
aws sesv2 get-account --query '{ProductionAccessEnabled:ProductionAccessEnabled,SendingEnabled:SendingEnabled}'

CloudTrail

Compare SetQueueAttributes and UpdateEventSourceMapping management events with the first failure. They can reveal a changed redrive policy or trigger. Worker logs provide per-message errors; ordinary Event history is not a record of every queue receive.

Fix

For this error, verify the intended test recipient and sender; production sending needs SES production access. Repair schema, permission or capacity errors before a controlled redrive. The source’s maxReceiveCount is 3; raising it only delays a permanent failure.

Prevent

Keep the DLQ alarm and partial batch reporting. Test poison messages and SES rejection. Confirm the affected consumer succeeds after repair; an alarm changing to OK is not proof of delivery.

apps/worker/src/batch.ts → handleBatch; Inspect the email DLQ →.

Say it in 30 seconds

Say it like this (20-30 seconds)

A growing dead-letter queue means messages exhausted their allowed receives, not necessarily that the payload is bad. I match a job to its worker error. For an SES verification error I check sandbox status and sender and recipient identities. I fix that cause before redriving, because retries cannot repair verification. Our queues allow three receives, isolate failed messages with partial batch responses, and alarm when the email DLQ is nonempty.

Advanced: Backlog, failure and retention are different

Email failures request delays of 10, 40 and 120 seconds; tickets rely on their 180-second queue visibility. Both source queues retain messages for four days and DLQs for fourteen. On standard queues, retention age still starts when the message was originally sent. The email DLQ alarm uses missing-data ignore; check the queue directly before deciding it is empty. The source queue’s oldest-message-age alarm covers stalled consumption separately. infra/src/jobs/queues.ts → createJobQueues; infra/src/observability/alarm-specs.ts. AWS explains SQS/Lambda failures and retries. SES lists verification errors and their Region scope.