The dead-letter queue is growing. How would you recover without making the incident worse?

Say it in 30 seconds

Say it like this (20-30 seconds)

I first identify the failing operation and stop treating retries as the fix. I correlate a few booking IDs with worker logs, distinguish a permanent bad payload from a temporary dependency failure, and repair the cause. Then I redrive a controlled amount and verify business outcomes plus the source queue. MiniTickets has duplicate suppression, but email sends can still repeat, so an empty DLQ alone is not proof of successful recovery.

The deeper answer

A dead-letter queue retains messages after repeated delivery failures. It preserves evidence and gives an operator time to investigate; it does not repair a missing IAM permission, an unsupported schema or an unverified SES recipient.

MiniTickets configures three receives before redrive to a DLQ, four-day retention in the source queues and fourteen days in their DLQs. I would check when the oldest affected messages were enqueued because retention can expire evidence and work. A message already too old is not rescued by a healthy alarm.

The local redrive implementation sends to the source queue before deleting from the DLQ. A crash between those operations can duplicate the message, but the reverse order could lose it. In production, recovery should include a small sample of actual email or ready-ticket outcomes, failure rate after redrive and a plan for permanently invalid events. Replaying every bad message immediately can amplify a downstream outage.

Counter-questions

Can you inspect SQS without changing anything?

Queue attribute reads are read-only. Receiving message bodies affects visibility and receive counts; the repository listing script also restores visibility. Treat that as an operational action, not a passive database query.

What if SES accepted the email before the failure?

A failed completion write can still put that booking through retries and into the DLQ. Redrive may send another email, so inspect the per-booking outcome before assuming every message represents an unsent email.

Why verify the source queue too?

Redrive moves work; it does not complete it. The DLQ can empty while the source queue accumulates retries, ages out messages or sends them back to the DLQ.

Where in MiniTickets

Read Sqs and dlq, then inspect the jobs-dlq piece →.

infra/src/jobs/queues.ts → createJobQueues

Use the DLQ runbook and growing-DLQ case.