A migration failed in the middle of a deploy

Symptom

Diagnostic scenario: CD stops at Run the database migration and prints the script’s actual failure text: “ERROR: migration failed. Read the log lines above.” A SQL failure also logs “migration failed:”.

Read the error first

The shell script reports the ECS container’s failure, not its underlying cause. Read the stopped reason, exit code and SQL error. Reach means the database cannot be reached; Allowed may mean execution-role secret access; Password means database authentication. Invalid SQL is a fourth possibility: the database received and rejected the statement.

Decide

flowchart TB
  A[Migration step failed] --> B{Container has exit code?}
  B -->|No| C[Read stopped reason: image or secret]
  B -->|Yes| D{Database connection established?}
  D -->|No| E[Check reach and authentication]
  D -->|Yes| F[Read first SQL error and transaction]
  F --> G[Fix forward before phase B]

Likely causes

Ranked for triage, not measured incident frequency:

1. SQL rejected a constraint, data shape or statement; inspect the first database error.
2. The migration image or secret could not be retrieved; there may be no container exit code.
3. The task cannot reach Postgres or its injected database credentials are wrong.

How to check

Console

GitHub → failed migration step: capture its task id and last 20 log lines. ECS → cluster → stopped task: inspect container exit code and stopped reason. CloudWatch → /minitickets/dev/api → migrate/migrate/ stream for that task: find the first error.

CLI

export AWS_PROFILE=ticketlite
aws ecs list-tasks --cluster minitickets-dev --family minitickets-dev-migrate --desired-status STOPPED
aws ecs describe-task-definition --task-definition minitickets-dev-migrate --query 'taskDefinition.{image:containerDefinitions[0].image,command:containerDefinitions[0].command}'
aws logs tail /minitickets/dev/api --log-stream-name-prefix migrate/migrate/ --since 1h

Use the returned task ARN with ECS task details to inspect that specific run. These commands do not run another migration.

CloudTrail

Find RunTask under the CD role session and inspect requested task definition and time. CloudTrail records ECS control calls, not SQL statements; use the task’s CloudWatch logs for the database failure.

Fix

Correct the migration or its prerequisite and deploy the reviewed fix. Inspect transaction outcome before deciding what to rerun. Phase A infrastructure can remain changed even though phase B never ran.

Prevent

Keep one migration task and the exit-code gate in scripts/run-migrations.sh. Test pending migrations against representative data. Use compatible schema changes because existing app tasks keep serving against the migrated schema.

Say it in 30 seconds

Say it like this (20-30 seconds)

I first distinguish a container that never started from a migration process that exited with an error. Then I read its first database error and establish whether the transaction committed. MiniTickets stops before phase B on failure, but phase A infrastructure changes can remain. I correct the specific cause and deploy a reviewed fix. Backward-compatible migrations protect the old app while schema changes happen.

Advanced: Transaction and first-deploy limits

apps/api/src/db/migrate.ts uses Drizzle’s Postgres migrator. The installed implementation applies pending migrations in one transaction, so a failed statement rolls that run back. This is not a general claim about every migration tool or external side effect. On an existing deployment phase A normally preserves the current service image. If there is no current tag, CD starts services with the selected image in phase A; it cannot preserve an old app that does not exist. A worker bootstrap can also use the new worker image early.