All runbooks

Start every incident with the first five minutes. Then open the runbook that matches. Each one has four parts: Signal (what you see), Check (read-only first), Fix and Verify. Write actions are described as console clicks, commits or CD runs, never as commands to paste. Placeholders like <cluster> come from the stack outputs.

Alarms

DLQ has messages

• Signal: alarm minitickets-dev-jobs-dlq-not-empty. At least one email job exhausted retries; SES may already have accepted the email before recording completion failed.
• Check: Logs Insights → saved query minitickets-dev/errors; take a correlationId and run minitickets-dev/follow-one-request. Count: aws sqs get-queue-attributes --queue-url <jobs-dlq-url> --attribute-names ApproximateNumberOfMessages.
• Common causes: SES sandbox (“Email address is not verified”), DKIM records missing, a code change that rejects the message shape. See SES and SQS and the DLQ.
• Fix: fix the cause first (verify the recipient, fix DNS, or deploy a fix). Then SQS console → minitickets-dev-jobs-dlq → Start DLQ redrive → redrive to the source queue.
• Verify: track each affected booking until the worker logs “email sent” or “skipped, already done”. SES acceptance does not prove inbox delivery. Confirm the source queue drains too: redrive alone can empty the DLQ and clear its alarm before processing succeeds. Completed-job records suppress repeats while retained (30-day TTL); send-before-record crashes or concurrent copies can still send duplicate emails.

Jobs queue is stuck

• Signal: alarm minitickets-dev-jobs-oldest-message-age (a message waited 5 minutes or more).
• Check: is computeEnabled true in the committed infra/Pulumi.dev.yaml? Then aws lambda list-event-source-mappings --function-name minitickets-dev-worker (State must be Enabled), and the dashboard’s worker graph for Throttles.
• Fix: compute off → turn it on with a commit (see “Turn the environment off or on”). Trigger disabled by hand → Lambda console → minitickets-dev-worker → Configuration → Triggers → enable it. A normal CD apply does not notice a change made outside the code. Throttles → the account’s Lambda limit may be 10; request an increase in Service Quotas.
• Verify: the oldest-message age falls to seconds; the alarm returns to OK.

Worker crashes or times out

• Signal: alarm minitickets-dev-worker-errors.
• Check: saved query minitickets-dev/worker-durations (max against the 10 s timeout) and minitickets-dev/errors; Lambda → minitickets-dev-worker → Monitor.
• Fix: a slow SES or DynamoDB call: undeleted messages become visible when their visibility expires (normally 60 s, but earlier per-message backoff can change it); watch. A code bug in the last deploy: roll back the app.
• Verify: Errors back to 0 for 5 minutes; no new messages in the DLQ.

Visitors get 5xx

• Signal: alarm minitickets-dev-alb-5xx (5 or more in 5 minutes), or users report errors.
• Check: the dashboard separates target 5xx (the app returned a 5xx response) from ALB-generated 5xx. Inspect the affected route: /trpc/* goes to API; pages go to web. Run aws ecs describe-services --cluster <cluster> --services <affected-service> for service events, and aws elbv2 describe-target-health --target-group-arn <affected-target-group-arn>. ALB 502 can mean a reset or malformed response; 503 can mean no registered or usable targets; 504 can mean a connection or response timeout. See AWS’s error guide.
• Fix: check replacement task events after a Spot interruption; investigate repeated start or health-check failures. A bad deploy: roll back the app. For target 5xx, follow the request’s logs into Postgres or Valkey; /health does not check dependencies.
• Verify: request the failing route again, check replacement tasks and target health, and confirm both 5xx metrics settle and the alarm returns to OK. No traffic alone can produce OK.

Alarm emails do not arrive

• Signal: an alarm shows ALARM in the console, but nobody got an email.
• Check: aws sns list-subscriptions-by-topic --topic-arn <alert-topic-arn>; the email subscription must not be “PendingConfirmation”.
• Fix: click “Confirm subscription” in AWS’s email to alertEmail (check spam). If that email is lost, SNS console → the subscription → Request confirmation sends a new one.
• Verify: SNS console → the topic → Publish message → a test email arrives.

Deploys

Roll back the app

• Signal: the last deploy broke the app (5xx, errors in the logs right after it).
• Check: the previous good tag: ECR → minitickets-dev-api → images tagged deployed-*, or the summary of the previous CD run. The old version must work with the current database schema.
• Fix: GitHub → Actions → CD → Run workflow on main, image_tag = that SHA, allow_older_infra unticked. Approve the deploy job.
• Verify: the run’s summary; ECS service running the new revision with the old image; the smoke test passes. Then fix forward in a new commit. See rollback.

Deploy guard refused

• Signal: CD fails at “Deploy guard: refuse old or superseded infrastructure code”.
• Check: the ::error:: line. “Superseded”: main has a newer commit. “OLDER than the deployed commit”: this commit is behind /minitickets/dev/deployed-commit.
• Fix: superseded → approve the newest run instead. Older and you wanted an app rollback → use image_tag. Only for a deliberate infrastructure rollback, a manual run with allow_older_infra: it deletes what newer commits added.
• Verify: the new run passes the guard step.

Migration failed

• Signal: CD stops at “Run the database migration”. The app still runs the old version.
• Check: the step’s last 20 log lines, or CloudWatch → /minitickets/dev/api, streams starting with migrate/. All SQL files of that run were rolled back together.
• Fix: correct the migration in a new commit, push, let CI pass, approve the new deploy. Infrastructure changes from phase A stay applied.
• Verify: the migration step exits 0 and phase B rolls. See migrations.

Data and cost

Database password rotated

• Signal: the api log shows “password authentication failed” for every query.
• Check: Secrets Manager → the RDS master secret → Rotation (last rotated date).
• Fix: running tasks hold the old password. ECS → the cluster → the api service → Update → Force new deployment. New tasks read the current secret at start.
• Verify: the errors stop; the service is stable. See secrets.

Turn the environment off or on

• Signal: a work session ends (off), or starts (on). Or a budget email arrives.
• Check: the committed switches in infra/Pulumi.dev.yaml.
• Fix: off: set entryEnabled, computeEnabled and dataEnabled to false (data off deletes the database). On: data, then compute, then entry, or all at once. Commit, push, approve the CD deploy.
• Verify: the CD summary; Cost Explorer the next day. See cost switches.