What do you do in the first five minutes of a booking incident?

Say it in 30 seconds

Say it like this (20-30 seconds)

I establish customer impact and timing, check recent changes, and split the request path from asynchronous work. MiniTickets can accept bookings while confirmations fail, so I verify database booking outcomes separately from email and tickets. I use the dashboard, deployment history and correlation IDs to narrow the failing stage. Then I choose a bounded mitigation, preserve evidence and verify recovery against the affected outcome before investigating the full root cause.

The deeper answer

First identify scope: all requests, one route, one event or only background completion. Record the earliest known failure and a representative correlation or booking ID. Avoid asking customers to repeat bookings with fresh idempotency keys before checking whether the first request committed.

For public errors, distinguish ALB-originated 5xx from application errors, inspect target registration and ECS events, and compare deployed revisions. For delayed confirmations, inspect outbox progress, SNS delivery, queue age, Lambda errors and DLQ evidence. The current dashboard focuses on API and email; ticket work needs its own logs and resource checks.

A recent release is a hypothesis, not proof. A compatible image rollback may be a useful mitigation; it cannot undo a committed booking or schema change. During a larger incident, give one person coordination and status duties so investigation and communication proceed together. Recovery means a real affected path works and delayed work is resolving, not merely that an alarm changed state.

Counter-questions

Would you purge the queue to make the alarm green?

No. That discards customer work and destroys evidence. Diagnose the failure and choose a reviewed retry or repair path.

What if you cannot safely roll back?

Reduce the affected operation or repair forward while preserving durable state. Compatibility determines the available mitigations.

When do you write the root-cause report?

After service is stable and evidence is preserved. Record the timeline during response, then separate proven causes from hypotheses afterward.

Where in MiniTickets

Read the path chapter, select the relevant map piece →, then inspect infra/src/observability/dashboard-body.ts → dashboardBody. Work through the related case for a diagnostic example.