Runbooks

In 1 minute

A runbook is a short, written checklist for one known situation: what you see, what to check first, how to fix it, and how to know it worked. It is written on a calm day, so nobody has to think from zero at 2 a.m. Every MiniTickets incident starts with the same five minutes, then the matching runbook from the list of all runbooks.

Picture it

flowchart TB
  A["alarm email or a user report"] --> I["1. is a customer affected?<br/>dashboard"]
  I --> L["2. what fails?<br/>errors query, correlation id"]
  L --> C["3. what changed?<br/>last CD run, ECS events"]
  C --> D{"4. decide"}
  D --> F["fix forward"]
  D --> R["roll back (image_tag)"]
  D --> W["contain and wait<br/>(messages stay queued)"]

How it works

1. Minute 1: read the alarm. Its name says what is wrong; its description says where to look first. Note the time.
2. Minute 2: is a customer affected? The dashboard’s alarm box and the graph for that part. A DLQ message is one customer without an email; ALB 5xx is many visitors with errors.
3. Minute 3: what fails? Logs Insights, saved query minitickets-dev/errors. Take one correlationId and run follow-one-request.
4. Minute 4: what changed? The last CD run in GitHub Actions, and the ECS service events. Most incidents follow a change.
5. Minute 5: decide. Fix forward if the cause is clear and small. Roll back the app if the last deploy caused it. Or contain: queued messages wait up to 4 days, so an email problem can wait for a proper fix.

In MiniTickets

Situation	Runbook
jobs-dlq-not-empty	DLQ has messages
jobs-oldest-message-age	Jobs queue is stuck
worker-errors	Worker crashes or times out
alb-5xx	Visitors get 5xx
A bad deploy	Roll back the app
CD refused or stopped	Deploy guard refused, Migration failed

Each runbook has the same four parts: Signal, Check (read-only first), Fix, Verify. Write actions are described as console clicks or CD runs, never as commands to paste: a pasted write command in a hurry is how incidents grow.

Play the failing email: it ends with this checklist →

See it in AWS

Console

CloudWatch → Dashboards → minitickets-dev, then Logs Insights → Saved queries.

CLI

aws cloudwatch describe-alarms --state-value ALARM --alarm-name-prefix minitickets-dev-

Code

Each alarm’s description, with where to look: infra/src/observability/alarm-specs.ts → alarmSpecs.

Break it

Symptom	Cause	Where to look
Nobody noticed an incident for hours	The alert subscription was never confirmed	SNS → alerts topic → Subscriptions
A “fix” made things worse	A write command run in a hurry, against the wrong thing	CloudTrail event history
Two people fix the same thing at once	Nobody said “I have it”	The incident channel

Say it in an interview

Say it like this (20-30 seconds)

A runbook is a written checklist for a known situation: signal, read-only checks, fix, and how to verify. Our first five minutes are always the same. Read the alarm and its description, check the dashboard for customer impact, run the saved errors query and follow one correlation id, look at the last deploy and ECS events, then decide: fix forward, roll back the app by image tag, or contain.

Follow-ups

Why read-only checks first?

They cannot make things worse, and they often show that the cause is somewhere else than it seemed. Writes come after the cause is known.

When do you roll back instead of fixing forward?

When the last deploy clearly caused it and the old version is compatible with the current schema. A rollback is a known state; a quick fix is a new, untested one.

What happens after the incident?

A short write-up without blame: what happened, why, and one check, alarm, test or rule that stops it next time. Then update the runbook.

Advanced: Details: who did what

CloudTrail’s event history (90 days, free) shows every AWS API call with the role and session name. CD sessions are named after the GitHub run id, so a change can be traced to one run.

Checkpoint

Answer these without scrolling up. If one feels hard, reread that section.

1. What are the four parts of every runbook?
2. What do you do in each of the first five minutes?
3. Which saved query do you run first?
4. When is “contain and wait” a good choice?
5. Why are write actions never written as commands to paste?

Redraw it from memory: alarm → customer affected? (dashboard) → what fails? (errors query, correlation id) → what changed? (CD run, ECS events) → fix forward / roll back / contain

Try this: on the map, play the failing-email flow to the end and name the runbook the last step needs

Mark this page as read