The two-phase deploy

In 1 minute

New code may need new infrastructure (a queue, a permission) and a new column. Rolled out first, it would fail until those exist. So the deploy job has two phases. Phase A applies all infrastructure changes and migrates the database while the services keep running the current version. Phase B switches the services to the new version and then checks that it really runs.

Picture it

flowchart TB
  G["guard + checks"] --> A["phase A: pulumi up<br/>infra changes, migration → new image,<br/>services keep current tag"]
  A --> M["migrate (new image)"]
  M --> B["phase B: SSM image-tag = new,<br/>tag deployed-sha, pulumi up → roll"]
  B --> V["verify: revisions, Lambdas,<br/>smoke test"]

How it works

1. Checks. The guard, the tag in all three repositories, no committed image-tag override. With computeEnabled: false committed, the job only applies the config and stops.
2. Phase A. An infrastructure apply with the migration image set to the new tag. The services read the current tag from SSM /minitickets/dev/image-tag, so they do not change.
3. Migrate. One Fargate task runs the new SQL files. A failure stops here.
4. Phase B. SSM image-tag becomes the new tag, each image gets a deployed-<sha> tag, and a second apply gives the task definitions new revisions. ECS rolls each service: new tasks first, old ones stop once the new ones are healthy.
5. Verify. Wait until the services are stable, compare each running task definition with the one just registered, and check both Lambdas and their queue triggers. With entry on, / and /trpc/events.list must answer 200.

In MiniTickets

“Stable” is not “deployed”: if new tasks never get healthy, ECS’s circuit breaker rolls back to the old revision, and that is stable too. Comparing revisions catches it.

First deploy. If SSM still says none, or the current image is gone, there is no version to keep: the services start with the new tag in phase A. The worker Lambdas also move early when the current tag lacks a worker image or the ticket handler; they never use SQL.

The smoke test makes up to 12 tries, 10 s apart. In ALB-only mode it connects straight to the ALB (curl --connect-to) while asking for our domain, so it works before public DNS points there.

This follows a normal deploy with compute on: CI tests a commit, CD publishes its images, then migrates and rolls the services. GitHub environment dev waits for approval when its required reviewer is configured; compute off only applies the committed infrastructure config.

1. CI workflow → CD publish job. CI finished green on main, so workflow_run starts CD.If it breaks: CI failed: CD does nothing. Look: GitHub → Actions → CI → the failed job.
2. CD publish job → GitHub publish role. The job swaps its GitHub OIDC token for 1-hour credentials. The sub must be exactly …:ref:refs/heads/main.If it breaks: The sub does not match: Not authorized to perform sts:AssumeRoleWithWebIdentity. Look: The role's trust policy versus the job's sub claim.
3. CD publish job → ECR repositories. It builds api, web and worker for ARM64, tags them with the git SHA and pushes them (existing tags are skipped).If it breaks: A brand-new repository is not in the role's policy yet. Look: The publish job log; IAM → minitickets-dev-gh-publish → push-images.
4. CD publish job → CD deploy job. The deploy job uses environment dev; with a required reviewer configured there, it waits for a person to click Approve.
5. CD deploy job → GitHub deploy role. It assumes the deploy role with sub …:environment:dev, for up to 2 hours.
6. CD deploy job → Release parameters. The guard refuses older infrastructure unless a manual infra rollback allows it; CI-started runs also stop if main has moved on.If it breaks: An old approved run is refused before it can delete newer resources. Look: The guard's message; aws ssm get-parameter --name /minitickets/dev/deployed-commit.
7. CD deploy job → Migration task definition. Phase A applies infrastructure and gives the migration the new image; run-migrations.sh uses FARGATE. Services normally keep the current tag; bootstrap promotes them early if that tag is none or missing, and promotes workers if their image or ticket handler is missing.
8. Migration task definition → RDS Postgres. node dist/migrate.js applies the new SQL files.If it breaks: A migration fails: CD stops before phase B; infrastructure changes and bootstrap tasks from phase A may already exist. Look: The migrate/migrate/<task id> stream in /minitickets/dev/api, printed by run-migrations.sh.
9. CD deploy job → Release parameters. Phase B: /minitickets/dev/image-tag = the new SHA, and the images get the tag deployed-<sha>.
10. Release parameters → API task definition. pulumi up reads the tag: new task definition revisions with the new image.
11. API ECS service → API task definition. The services roll to the new revisions. The circuit breaker rolls back if new tasks never get healthy.If it breaks: Rolled back: the old version keeps running, and cd.yml notices. Look: ECS → minitickets-dev-api → Events; cd.yml prints the latest ECS service events.
12. CD deploy job → API ECS service. CD checks service revisions and both Lambdas; with entry on, GET / and /trpc/events.list must return 200. Once any apply starts, the final recording step records the infrastructure commit even after failure or cancellation.If it breaks: A smoke-test endpoint does not return 200: CD prints the ECS service events and fails. Look: The Smoke test output and 'Print the latest ECS service events' step; the ALB group's inbound rules.

Play the deploy on the map →

See it in AWS

Console

ECS → Clusters → the cluster → Services → api → Deployments; GitHub → Actions → CD → job deploy → summary.

CLI

aws ssm get-parameter --name /minitickets/dev/image-tag shows the tag phase B wrote.

Code

.github/workflows/cd.yml → deploy.

Break it

Symptom	Cause	Where to look
“Wait for the services to be stable” fails; the old version runs	The circuit breaker rolled back	“Print the latest ECS service events”
The smoke test fails	An app error, an unhealthy task, or allowedCidrs blocks the runner	The step’s tries
The worker check fails: trigger not Enabled	Nothing reads the jobs queue	Lambda → Triggers

Say it in an interview

Say it like this (20-30 seconds)

The deploy job has two phases. Phase A applies every infrastructure change and points only the migration task at the new image, while the services keep the current tag from SSM. Then the migration runs. Phase B writes the new tag to SSM, applies again, and ECS rolls the services. Finally we compare the running task definitions with the registered ones, because a circuit-breaker rollback also looks stable, and smoke-test the public endpoints.

Follow-ups

Why is the tag in SSM and not in the committed config?

The committed config says WHAT exists; SSM says WHICH version runs. CD writes SSM after the migration, and any local preview reads the same value, so nobody rolls the app back by accident.

What if phase A fails halfway?

Some infrastructure changes may exist, the app is unchanged, and the commit is still recorded as deployed, so an older run cannot delete what was created. Fix and deploy again.

Why restart the api when the web service is new?

Service Connect names are learned when a task starts. An api task started before web existed cannot call it, so CD forces one api restart.

Advanced: Details: overrides and timing

--config migrationImageTag=<tag> changes only the runner’s copy of the config, which is thrown away. CD refuses apiImageTag, webImageTag, migrationImageTag and workerImageTag in the committed file: they would override SSM. Turning everything on can take RDS 10-15 minutes, which is why the job may run 110 minutes.

Checkpoint

Answer these without scrolling up. If one feels hard, reread that section.

1. What changes in phase A, and what does not?
2. Where do the services read their image tag from?
3. Why is “stable” not enough to call a deploy done?
4. What happens on the very first deploy?
5. How does the smoke test reach a new ALB before DNS points at it?

Redraw it from memory: checks → phase A (infra + migration task → new image) → migrate → phase B (SSM tag, deployed tag, roll) → verify (revisions, Lambdas, smoke test)

Try this: on the map, play the deploy flow and open 'What if it breaks?' on the migration step

Mark this page as read