CD deployed old code

Symptom

Diagnostic scenario: users still see old behavior. The workflow’s revision check may report “ECS rolled the deploy back (deployment circuit breaker): the new tasks did not become healthy.” That is a source-defined error, not a newly observed incident.

Read the error first

The CD script reports that ECS is running a different task definition from the expected one. ECS being stable means its tasks have settled; they could belong to the previous release. Start with release state. Reach, Allowed or Password errors may explain a failed new task, but the old UI alone proves none of them.

Decide

flowchart TB
  A[Old behavior after deploy] --> B{Intended tag selected?}
  B -->|No| C[Check rollback input and run SHA]
  B -->|Yes| D{Phase B reached?}
  D -->|No| E[Inspect earlier failed step]
  D -->|Yes| F{Running revision matches?}
  F -->|No| G[Read task failures and rollback]
  F -->|Yes| H[Check DNS and cache layers]

Likely causes

Ranked by the evidence to inspect first, not measured frequency:

1. New tasks failed health checks and ECS restored the previous completed deployment.
2. Migration or another earlier step stopped deployment before phase B, or manual image_tag deliberately selected older images.
3. Tasks are correct, but the browser sees cached data or public DNS still reaches another ALB.

How to check

Console

Repository → Actions → CD → run → Summary: compare chosen tag and each phase result. ECS → Clusters → minitickets-dev → Services → api/web service: compare task definition revisions, tasks and Events. ECR → Repositories → Private → repository → Images: compare immutable tag and digest. Check public DNS separately; ALB-only smoke tests bypass it.

CLI

export AWS_PROFILE=ticketlite
aws ssm get-parameter --name /minitickets/dev/image-tag --query Parameter.Value
aws ecs describe-services --cluster minitickets-dev --services minitickets-dev-api minitickets-dev-web --query 'services[].{name:serviceName,revision:taskDefinition,events:events[:3]}'
API_REVISION=$(aws ecs describe-services --cluster minitickets-dev --services minitickets-dev-api --query 'services[0].taskDefinition' --output text)
aws ecs describe-task-definition --task-definition "$API_REVISION" --query 'taskDefinition.containerDefinitions[].image'

The last command inspects the exact API revision selected by its service. Repeat for web; inspect running tasks too during a rollout, when old and new revisions can coexist.

CloudTrail

Correlate UpdateService, RegisterTaskDefinition and SSM PutParameter with the run’s gh-deploy- session. These show requested changes; ECS service events and running tasks establish whether the rollout succeeded.

Fix

Fix the new task’s actual failure, then deploy a reviewed release. For a deliberate app rollback, select a compatible existing image with main’s current infrastructure. Check DNS if the public address differs from the smoke test target.

Prevent

Keep the revision comparison in .github/workflows/cd.yml. An SSM image tag is desired release state, not proof of running code. Preserve immutable tags and verify responses after checking tasks.

Say it in 30 seconds

Say it like this (20-30 seconds)

I compare the intended image, the SSM release tag, and each service’s actual task definition. Stable ECS tasks can be an automatic rollback, so waiting for stability alone is insufficient. MiniTickets checks the expected revision after waiting. If those revisions match, I check DNS and the cache path behind the user response. I fix the failing layer and verify both running code and public behavior.

Advanced: Why a release record can lead the rollout

CD writes the image tag before phase B applies the service change. A later failure can leave SSM naming a version that is not running. The separate deployed-commit record can also include a partial infrastructure apply; it is a stale-deploy boundary, not a success certificate. AWS describes circuit-breaker rollback to a previous completed deployment.