The dashboard

In 1 minute

A CloudWatch dashboard is one page of graphs. MiniTickets has one, minitickets-dev, laid out from the outside in: the alarms, the front door (the ALB), the api tasks, the job pipeline, the worker, and the newest error lines. When users say “slow”, you read it top to bottom and stop at the first graph that looks wrong. It is free: an account’s first three dashboards cost nothing.

Picture it

flowchart TB
  A["1. alarms: anything red?"] --> L["2. ALB: requests, 5xx,<br/>response time p50 / p95"]
  L --> C["3. api tasks:<br/>CPU and memory %"]
  C --> Q["4. jobs queue and DLQ"]
  Q --> W["5. worker: calls, errors,<br/>throttles, duration"]
  W --> E["6. newest error lines<br/>with correlationId"]

How it works

1. Alarms box. Every alarm in one widget, red when in ALARM. If one is red, start from its description.
2. The front door (entry on). Requests and 5xx per minute, split into “from api/web” (target 5xx) and “from the ALB” (502/503/504), and TargetResponseTime at p50 and p95.
3. The api tasks (compute on). CPU and memory in % of what the task definition reserves.
4. The job pipeline (always on). Messages waiting, in flight, the oldest age, and the DLQ, which should be 0.
5. The worker (compute on). Calls, errors, throttles, and duration p50 and max against the 10 s timeout.
6. Newest errors. A log widget with error lines from the api and the worker and their correlationId, the same query as the saved errors query.

In MiniTickets

The body is built in code from the alarm ARNs, queue names and service names. A part for a layer that is switched off is simply left out, so with entry off there are no ALB graphs. A text box at the top says where to start: find the alarm, then Logs Insights, then follow one correlation id.

Typical causes of “slow” in MiniTickets: the one api task on Fargate Spot was just replaced and the new one is still starting; bookings wait on the row lock of a very popular event; Valkey answers slowly, so caches fall through to Postgres. The dashboard shows the first; the logs and the database’s own metrics show the others.

See the dashboard on the map →

See it in AWS

Console

CloudWatch → Dashboards → minitickets-dev. Set the time range to the last hour first.

CLI

aws cloudwatch get-dashboard --dashboard-name minitickets-dev

Code

infra/src/observability/dashboard-body.ts → dashboardBody and infra/src/observability/dashboard.ts → createDashboard.

Break it

Symptom	Cause	Where to look
The ALB graphs are missing	Entry is off, so those widgets are left out on purpose	Committed entryEnabled
An old layout after a change to the body code	The body changes only with an infrastructure apply, which CD runs after approval	The last CD run
The worker graphs are flat while emails are late	Nothing calls it: the queue trigger is disabled	Lambda → Triggers

Say it in an interview

Say it like this (20-30 seconds)

I start at our one CloudWatch dashboard, read from the outside in. First the alarm box. Then the ALB: request count, 5xx split by source, and response time at p50 and p95. Then the api tasks’ CPU and memory, then the jobs queue and its DLQ, then the worker’s errors, throttles and duration. The bottom widget shows the newest error lines with their correlation ids, so I can follow one slow request through the logs.

Follow-ups

Why p95 and not only the average?

An average hides the slow few. p95 is the time that 95% of requests beat; a jump there while p50 stays flat points to a subset, such as one endpoint or one task.

What does the dashboard not show?

Postgres and Valkey metrics, and the browser’s own timing. RDS and ElastiCache have their own metrics in their consoles; adding them is a choice against the dashboard’s metric limit.

Why build the dashboard in code?

It follows the switches, uses real resource names, and a review shows every change. A dashboard clicked together in the console drifts and is lost when the stack is rebuilt.

Advanced: Details: cost and limits

The first 3 dashboards of an account are free, with up to 50 metrics each; ours has about 20. The log widget’s query text is shared with the saved queries, so they never drift apart (infra/src/observability/dashboard-body.ts → ERRORS_QUERY).

Checkpoint

Answer these without scrolling up. If one feels hard, reread that section.

1. In which order do you read the dashboard, and why that order?
2. What do target 5xx and ALB 5xx each mean?
3. What does a high p95 with a normal p50 tell you?
4. Why are some graphs missing when a layer is off?
5. Which widget links the graphs to the logs?

Redraw it from memory: alarms → ALB (requests, 5xx, p50/p95) → api CPU/memory → jobs queue + DLQ → worker → newest error lines

Try this: on the map, select the CloudWatch dashboard and see which pieces it reads from

Mark this page as read