Users get the same email twice

Symptom

Troubleshooting scenario: a customer gets two confirmations for one booking. Logs show “SES accepted email” twice for the same booking.confirmed:<bookingId> job, possibly with different sqsMessageId values. There need not be an exception.

Read the error first

SES accepted two sends; SQS provides at-least-once delivery, meaning repeats are possible. Under Reach / Allowed / Password, successful sends prove those calls reached SES and were allowed. A later DynamoDB failure may be Reach or Allowed; read its error. Missing allow, explicit deny and wrong password do not explain two successful sends by themselves.

Decide

flowchart TB
  A["Two emails, one booking"] --> B{"Completion recorded before second check?"}
  B -->|"no"| C{"Did sends overlap?"}
  C -->|"yes"| D["Concurrent check-then-send race"]
  C -->|"no"| E["Send succeeded, recording failed or crashed"]
  B -->|"yes"| F["Check job key, table and record lifetime"]
  D --> G["Preserve business key and document remaining gap"]
  E --> G

Likely causes

Ranked by the implementation’s known gaps; use the timeline to choose.

1. Email succeeds, then the process crashes or markDone fails; redelivery sends again.
2. Two workers both read “not done” before either records completion. The later conditional write cannot undo either send.
3. Wrong completion table/key, an expired completion record, or overlapping processing after an incorrectly shortened visibility timeout.

How to check

Console

Search /minitickets/dev/worker by job key and correlation id. Compare request ids, timestamps and sesMessageId; distinguish “email sent” from “skipped, already done”. Inspect the matching processed-jobs item privately; do not copy customer data into reports.

CLI

Set JOBS_QUEUE_URL from SQS and compare visibility with the 10-second function timeout.

export AWS_PROFILE=ticketlite
aws sqs get-queue-attributes --queue-url "$JOBS_QUEUE_URL" --attribute-names VisibilityTimeout
aws lambda list-event-source-mappings --function-name minitickets-dev-worker

CloudTrail

SetQueueAttributes can reveal a changed visibility timeout. DynamoDB item-level events require data-event logging; Event history alone cannot reconstruct the completion writes. The worker’s SES acceptance and completion logs are the first evidence.

Fix

Restore the business key, consistent read and completion store if broken. Keep send before record: reversing them can lose email after a crash. Current code accepts the remaining crash and concurrency windows; it does not guarantee exactly-once delivery.

Prevent

Test sequential duplicate suppression and send/record failures. Preserve 60-second email queue visibility. A future lease could reduce concurrent sends, but cannot atomically commit with SES.

apps/worker/src/handlers/booking-confirmed.ts → handleBookingConfirmed; Inspect completion records →.

Say it in 30 seconds

Say it like this (20-30 seconds)

Our consumer checks completion by booking id, sends the email, then records completion. That skips ordinary redeliveries, but two workers can both check before either records, and a crash after sending can leave no completion record. The conditional write protects the record, not the external send. I compare both send timelines and repair any key or visibility mistake, while explaining that exactly-once email is not guaranteed.

Advanced: What the completion record guarantees

DynamoProcessedJobs.isDone uses a strongly consistent read. markDone conditionally writes only when jobKey does not exist; it remembers completion for thirty days using DynamoDB TTL. TTL deletion is asynchronous. A retained expired item can still suppress a duplicate; once deleted, replaying an old booking can send again. Using SQS message id as the key would miss two publishes of the same booking. The outbox chapter explains that producer side of duplication. apps/worker/src/processed-jobs.ts → DynamoProcessedJobs.