Our incident: a stale approval deleted resources

Symptom

Historical incident, 2026-10-15: approving the waiting CD run for 6ea9575 deleted ten resources added by later commits. Current guard refusals include “Superseded: main already has a newer commit”. The original deletion did not produce that guard error; the guard was added afterward.

Read the error first

Pulumi successfully reconciled AWS with the old program it was given. This was neither Reach, Allowed nor Password failure: the approved deploy role was allowed to delete resources. The missing check was whether this infrastructure source was still safe to apply.

Decide

flowchart TB
  A[Unexpected resource deletions] --> B{Run used older source?}
  B -->|Yes| C[Reject other stale approvals]
  B -->|No| D[Inspect intended config and diff]
  C --> E[Preview current main]
  E --> F{CD prerequisites removed?}
  F -->|Yes| G[Owner restores prerequisites first]
  F -->|No| H[Review newest deploy for recovery]

Likely causes

Ranked for this recorded event:

1. Witnessed cause: an old approval applied old infrastructure after newer resources existed.
2. A waiting pre-guard workflow retained its old workflow definition; current protections cannot retrofit it.
3. A deliberate allow_older_infra override or incorrect switches can still authorize destructive changes today.

How to check

Console

GitHub → approved run: inspect exact SHA and start time. Compare newer runs and Pulumi’s update history. Systems Manager → Parameter Store → /minitickets/dev/deployed-commit identifies the recorded infrastructure boundary, not necessarily a fully successful deployment.

CLI

export AWS_PROFILE=ticketlite
git log -5 --oneline
aws ssm get-parameter --name /minitickets/dev/deployed-commit --query Parameter.Value

From infra/, run plain pulumi preview against reviewed current source. Inspect every proposed creation, update and deletion; historical counts do not predict today’s recovery diff.

CloudTrail

Correlate destructive API calls, such as DeleteFunction or DeleteQueue, with the gh-deploy- session and update time. These are events to look for, not a claimed inventory of the historical ten deletions.

Fix

Reject remaining obsolete approvals. The historical recovery restored publish/deploy policy and worker-repository prerequisites before CD recreated the remaining graph. Today, review the actual damage and have the owner repair prerequisites using current source and committed config.

Prevent

scripts/deploy-guard.sh rejects superseded CI runs and source older than the recorded commit. CD fetches the newest guard before infrastructure work. App rollback selects an old image with current infrastructure. Keep the ancestry regression tests.

Say it in 30 seconds

Say it like this (20-30 seconds)

An old approved run applied infrastructure code from before ten newer resources existed. Pulumi treated their absence from that program as instructions to remove them. Approval established permission, not freshness. We added a guard comparing the candidate with main and the recorded infrastructure commit before applying changes. Application rollback now selects older images with current infrastructure. We also reject waiting runs that predate the safeguard.

Advanced: What the guard cannot promise

CD serializes deploys per environment. Its record step attempts to save the source commit even after a started phase A fails or is cancelled, because infrastructure may already have changed. Abrupt runner loss or a failed record write can still leave that record behind reality. Missing SSM parameters mean no previous record; other read failures refuse deployment. Unknown history refuses unless deliberately overridden. A manual run skips the superseded-main check, but still checks recorded ancestry. The guard cannot reject a destructive edit in current source. .github/workflows/cd.yml retains allow_older_infra for deliberate infrastructure rollback; it can delete later additions. Frozen docs/34-approved-delivery.md preserves the event and local eleven-test measurement, not a new live acceptance claim.