What makes an alarm actionable rather than noisy?

Say it in 30 seconds

Say it like this (20-30 seconds)

An actionable alarm names an affected outcome, an owner and a first diagnostic step. MiniTickets watches the email DLQ, jobs queue age, worker invocation errors and combined ALB 5xx counts. Those signals cover different failures; a successful Lambda invocation can still contain failed messages. I would tune production alerts against customer impact and expected traffic, then verify notification delivery and the runbook, rather than treating a green dashboard as a reliability guarantee.

The deeper answer

The jobs-age threshold is 300 seconds. A growing queue can mean no consumer, throttling, slow downstream work or incoming load beyond completion capacity. The alarm signal does not by itself prove the cause.

The worker-errors alarm watches invocation-level errors. The handler catches message failures and returns partial failures, so failed emails can reach the DLQ while Lambda Errors remains zero. A DLQ message also does not prove no email was sent: a send may succeed before its completion record fails.

The ALB alarm adds target and load-balancer 5xx counts, firing at five errors in five minutes. That is a development threshold; a production rate or error-budget signal needs a request denominator and a defined objective. Missing-data behavior and confirmed SNS email subscription matter as much as the threshold. Ticket-consumer-specific SLOs and alerts are not implemented by these four email-oriented alarms.

Counter-questions

Would you page on every failed message?

Use customer impact and urgency. A recoverable retry may need a metric; a growing DLQ or missed completion deadline may need a person.

Does an OK notification prove recovery?

No. Verify the affected request or business outcome. No traffic or changed metric reporting can also clear an alarm.

What would you add first in production?

Define booking success and asynchronous completion objectives, then close the highest-risk detection gap with an owner and runbook.

Where in MiniTickets

Read the path chapter, select the relevant map piece →, then inspect infra/src/observability/alarm-specs.ts → alarmSpecs. Work through the related case for a diagnostic example.