Alarms

In 1 minute

Nobody watches graphs all day, so something must tell a person when to act. A CloudWatch alarm watches one metric, compares it with a threshold every period, and is OK, ALARM or INSUFFICIENT_DATA. MiniTickets has four alarms, each answering “is a customer affected right now?” or “is something stuck?”. On every change to ALARM, and back to OK, the alarm publishes to an SNS topic that emails the owner.

Picture it

flowchart TB
  M["metrics: SQS, Lambda, ALB"] --> A["CloudWatch alarm<br/>threshold per period"]
  A -->|"ALARM / OK"| T["SNS topic<br/>minitickets-dev-alerts"]
  T -->|"confirmed subscription"| E["email to alertEmail"]

How it works

1. A metric arrives, usually every minute: messages in a queue, Lambda errors, 5xx answers.
2. The alarm evaluates one period: is the value above the threshold?
3. Missing data has a rule: notBreaching counts it as fine; ignore keeps the last state.
4. On a state change the alarm publishes to the alerts topic (alarmActions and okActions).
5. SNS emails the confirmed subscription. Each alarm’s description says where to look first.

In MiniTickets

Alarm	Fires when	Means	On when
jobs-dlq-not-empty	more than 0 messages in the DLQ, 1 minute	a customer got no email	always
jobs-oldest-message-age	a message waited 300 s or more	nothing reads the jobs queue	always
worker-errors	Lambda Errors above 0 in 5 minutes	the worker crashed or timed out	compute on
alb-5xx	5 or more 5xx in 5 minutes, ALB plus targets	visitors get server errors	entry on

Every name starts with minitickets-dev-. The topic policy lets cloudwatch.amazonaws.com publish only for alarms with that prefix in our account, and the deploy role may manage only those alarms.

The DLQ alarm uses ignore: SQS stops sending metrics for an idle queue after about six hours, and notBreaching would then flip it to OK while the message is still there.

From → to	What happens	What allows it
Jobs dead-letter queue → CloudWatch alarms	ApproximateNumberOfMessagesVisible > 0 for 1 minute → jobs-dlq-not-empty	metric AWS/SQS ApproximateNumberOfMessagesVisible, Maximum, period 60 s, treatMissingData ignorealarmSpecs
Jobs SQS queue → CloudWatch alarms	ApproximateAgeOfOldestMessage ≥ 300 s → jobs-oldest-message-age	metric AWS/SQS ApproximateAgeOfOldestMessage, Maximum, period 60 salarmSpecs
Worker Lambda → CloudWatch alarmsonly when computeEnabled = true	Errors > 0 in 5 minutes → worker-errors	metric AWS/Lambda Errors, Sum, period 300 salarmSpecs
Application Load Balancer → CloudWatch alarmsonly when entryEnabled = true	target 5xx + ALB 5xx ≥ 5 in 5 minutes → alb-5xx	metric AWS/ApplicationELB FILL(HTTPCode_Target_5XX_Count, 0) + FILL(HTTPCode_ELB_5XX_Count, 0), period 300 salarmSpecs
CloudWatch dashboard → CloudWatch alarms	the dashboard's alarm status widget	config dashboard body built from the alarm ARNs, queue names and service namescreateDashboard
CloudWatch alarms → Alerts SNS topic	each alarm publishes on ALARM and on OK	resource-policy topic policy OurCloudWatchAlarmsMayPublish: cloudwatch.amazonaws.com may sns:Publish, only for aws:SourceArn alarm:minitickets-dev-* in our accountcreateAlertTopic config alarmActions and okActions = [alert topic ARN]createAlarms

Play the failing email until the alarm fires →

See it in AWS

Console

CloudWatch → Alarms → All alarms, filter minitickets-dev. SNS → Topics → minitickets-dev-alerts → Subscriptions: Status must be Confirmed.

CLI

aws cloudwatch describe-alarms --alarm-name-prefix minitickets-dev-

Code

infra/src/observability/alarm-specs.ts → alarmSpecs.

Break it

Symptom	Cause	Where to look
An alarm is in ALARM, but nobody got an email	The email subscription is still Pending confirmation	SNS → Subscriptions
Emails fail, yet worker-errors stays OK	Failed messages are reported in batchItemFailures; the call itself succeeded	The DLQ alarm fires instead
alb-5xx fires during a deploy	A Spot interruption or slow start gave a few 502/503	ECS service events

Say it in an interview

Say it like this (20-30 seconds)

We alarm on four things. A message in the email dead-letter queue, because that is a customer without an email. The oldest message in the jobs queue waiting five minutes, because nothing is reading it. Worker Lambda errors, meaning a crash or timeout. And five or more 5xx answers in five minutes at the ALB. Each alarm publishes to an SNS topic whose policy allows only our alarms, and the topic emails the owner.

Follow-ups

Why five minutes for the queue age?

A healthy message leaves within seconds, and even a failing one leaves within about three minutes (tries, then the DLQ). Five minutes means stuck, not slow.

Why a count of 5xx, not a rate?

In dev, traffic is low, so a rate would swing wildly. A busy production site would alarm on 5xx divided by requests.

Why is the alerts topic not encrypted?

The free AWS-managed aws/sns key’s policy does not let CloudWatch use it, so alarms could not publish. A customer-managed key would work, for a monthly fee.

Advanced: Details: the 5xx sum

The alarm adds HTTPCode_Target_5XX_Count (our apps answered 500) and HTTPCode_ELB_5XX_Count (the ALB answered 502/503/504 itself) with metric math, FILL(m, 0) for periods without data (infra/src/observability/alarms.ts → createAlarms, infra/src/observability/alert-topic.ts → createAlertTopic).

Checkpoint

Answer these without scrolling up. If one feels hard, reread that section.

1. Name the four alarms and what each one means.
2. Which two alarms are always on, and why?
3. Why does the DLQ alarm treat missing data as ignore?
4. Why does a failed email not raise worker-errors?
5. What must happen once before any alarm email arrives?

Redraw it from memory: metric → alarm (threshold per period, missing-data rule) → SNS alerts topic (policy: our alarms only) → confirmed email

Try this: on the map, select the CloudWatch alarms and follow the arrow to the alerts topic; read its condition

Mark this page as read