Rollback and the stale-deploy guard

In 1 minute

There are two kinds of rollback. An app rollback runs an older image with today’s infrastructure: a manual CD run with image_tag set to an older SHA. An infrastructure rollback applies older infrastructure code, which deletes whatever newer commits added. The deploy guard refuses the second kind unless someone asks for it on purpose, and refuses runs that main has already moved past.

Picture it

flowchart TB
  R["deploy run for commit C"] --> S{"started by CI and<br/>main moved on?"}
  S -->|"yes"| X["refuse: superseded"]
  S -->|"no"| O{"C older than<br/>SSM deployed-commit?"}
  O -->|"yes, no allow_older_infra"| Y["refuse: older"]
  O -->|"no"| G["go on"]

How it works

1. App rollback. Actions → CD → Run workflow, image_tag = an older SHA. Publish is skipped. The run checks out main’s newest code, so the infrastructure stays current; only the image changes.
2. The guard runs first, before any infrastructure command. It reads SSM /minitickets/dev/deployed-commit, the commit whose infrastructure code CD applied last.
3. Rule 1, superseded (runs started by CI only): if main’s newest commit is not this run’s commit, refuse. Approve the newest run instead.
4. Rule 2, older: if this commit is an ancestor of the recorded one, its infrastructure code is older than what is live. Refuse, unless a manual run sets allow_older_infra.
5. Record. Once an infrastructure apply has started, the run records its commit, even if a later step fails or the run is cancelled.

In MiniTickets

The guard exists because of a real incident (2026-10-15). A deploy of commit 6ea9575 waited for approval for days. When it was approved, it applied that old infrastructure code, and Pulumi deleted 10 resources that newer commits had added. The approval page showed which commit, but nothing said “this one is old”.

The guard script is always read from main’s newest commit, never from the commit being deployed: an old commit may carry an old guard, or none. Recording failed runs matters for the same reason: phase A may already have created resources.

App rollbacks need compatible migrations: the old code runs on the newer schema. The images to roll back to are kept by the ECR rule for the last 10 deployed-* images.

From → to	What happens	What allows it
Release parameters → API task definitiononly when computeEnabled = true	pulumi reads image-tag, so the task definitions use image :<tag>	config readDeployedImageTag lists /minitickets/dev and picks image-tag (config apiImageTag would override it)readDeployedImageTag
CD deploy job → Release parameters	reads deployed-commit for the guard; normally writes image-tag in phase B (phase A if no current image); records deployed-commit at the end if an infrastructure apply ran, including failure or cancellation	iam deploy role OurReleaseParameters grants ssm:GetParameter and ssm:PutParameter on the two parameter ARNs; ReadOurParameterPath grants ssm:GetParametersByPath under /minitickets/devcreateCicd

See the release parameters on the map →

See it in AWS

Console

Systems Manager → Parameter Store → /minitickets/dev/deployed-commit → History.

CLI

aws ssm get-parameter --name /minitickets/dev/deployed-commit

Code

scripts/deploy-guard.sh (tested by scripts/deploy-guard.test.ts) and infra/src/release.ts → createRelease.

Break it

Symptom	Cause	Where to look
“Deploy guard: refuse old or superseded infrastructure code” fails	Older than the recorded commit, or main has a newer commit	The ::error:: line names which rule
A rollback stops at the image check	The old tag is no longer in all three repositories	ECR → images
A preview with compute on stops: no image tag	SSM image-tag is still none: CD never deployed	Parameter Store

Say it in an interview

Say it like this (20-30 seconds)

An app rollback is a manual CD run with an older image tag: it keeps main’s current infrastructure and only rolls the services back, so migrations must stay backward compatible. Before any apply, a guard compares the commit with the last applied one in SSM. It refuses commits older than that, and CI-started runs that main has moved past. We added it after an old approved run deleted ten newer resources.

Follow-ups

Why not just cancel old waiting runs?

GitHub keeps one waiting run per concurrency group, and it can still be old. A person can approve the wrong one. A server-side check against what is live catches both.

Why record the commit even when the run failed?

From the first apply on, AWS may hold this commit’s resources. If it were not recorded, an older run approved next would pass the guard and delete them.

Why SSM and not a git tag?

The deploy role cannot write to the repository, and AWS is the system of record for what is live. Two standard parameters are free.

Advanced: Details: the two parameters

Pulumi creates /minitickets/dev/image-tag and /minitickets/dev/deployed-commit with the value none and ignores later changes to the value, so an apply never writes none back. The deploy role may get and put exactly these two parameters (infra/src/release.ts → readDeployedImageTag).

Checkpoint

Answer these without scrolling up. If one feels hard, reread that section.

1. What is the difference between an app rollback and an infrastructure rollback?
2. Which two rules does the guard apply?
3. Where does the guard read the last applied commit from?
4. Why is the guard read from main’s newest commit?
5. What must be true of migrations for an app rollback to work?

Redraw it from memory: run for commit C → superseded? refuse → older than SSM deployed-commit? refuse (unless allow_older_infra) → go on → record C once an apply starts

Try this: on the map, play the deploy flow and open the failure branch of the guard step

Mark this page as read