ECS cannot pull the image

Symptom

Diagnostic scenario: ECS stops a new task with CannotPullContainerError. A related startup-error fragment is unable to pull secrets or registry auth under ResourceInitializationError. These are documented error categories, not a recorded MiniTickets outage.

Read the error first

ECS, the container scheduler, reports a failure before application startup. A missing manifest means the requested image is absent. An ECR access denial is Allowed: ECS uses the execution role during startup to pull images. A connection timeout is Reach. Registry authentication is separate from the app’s database Password.

Decide

flowchart TB
  A[Stopped task reason] --> B{Image reference exists?}
  B -->|no| C[Check repository and release tag]
  B -->|yes| D{Access denied?}
  D -->|yes| E[Check execution role]
  D -->|timeout| F[Check public IP, route and egress]
  D -->|image pulled| G[Check startup and architecture]

Likely causes

Rank within the matching error branch:

1. Missing image: wrong repository/tag, failed publish, or retention removed a needed image.
2. Timeout: missing public IP or internet route. MiniTickets has no NAT gateway or VPC endpoints.
3. Denied: execution role lost its ECR policy. An explicit deny needs its own investigation.
4. Pull succeeded, process fails: exec format error suggests an architecture mismatch; this is a startup failure, not a failed download.

How to check

Console

ECS → Clusters → minitickets-dev → Tasks → filter Stopped → task: inspect its reason, task definition and networking. Compare its image URI with ECR → Repositories → Private → minitickets-dev-api → Images. IAM → Roles → minitickets-dev-api-execution-role → Permissions should list AmazonECSTaskExecutionRolePolicy.

CLI

Replace task ARN and image tag with the failed task’s identifiers; these return metadata.

export AWS_PROFILE=ticketlite
aws ecs describe-tasks --cluster minitickets-dev --tasks <task-arn> --query 'tasks[].{Reason:stoppedReason,Containers:containers[].reason}'
aws ecr describe-images --repository-name minitickets-dev-api --image-ids imageTag=<image-tag> --query 'imageDetails[].{Digest:imageDigest,Tags:imageTags}'
aws iam list-attached-role-policies --role-name minitickets-dev-api-execution-role

CloudTrail

Check ECR BatchGetImage or GetAuthorizationToken errors when recorded, and IAM policy changes at that time. Compare userIdentity with the execution role. An absent event does not prove a successful pull; network failures may never reach ECR.

Fix

Repair the failed branch: restore the exact image through publishing, correct the execution-role grant, or restore public-subnet networking. Keep ARM64 images aligned with runtimePlatform in infra/src/compute/task.ts → createApiTask.

Prevent

CD checks image existence. Retention keeps the newest ten images tagged deployed-*, ranked by original push time; it does not protect every deployed image forever. Review that limit and the service’s public subnets and public IP together in infra/src/registry.ts → createRegistry and infra/src/compute/service.ts → createApiService.

Say it in 30 seconds

Say it like this (20-30 seconds)

I start with the stopped task reason because image pulls happen before the app runs. I compare the exact repository and tag with ECR, then separate access denied from a connection timeout. ECS uses the execution role to pull; the application’s task role is not the repair. In MiniTickets, I also check public IP and internet routing because there is no NAT. Finally, I verify ARM64 compatibility and retain deployed images for replacement tasks.

Advanced: What each role and error proves

infra/src/compute/iam.ts → createIamRoles attaches the AWS managed execution policy. The running app’s task role has a different job; adding ECR permissions there cannot fix ECS startup. API and web tasks use public IPs to reach ECR, Secrets Manager and Logs while their security groups still restrict inbound traffic.

A successful CLI read uses your laptop identity and network. It proves image existence for that identity, not that the task’s execution role or network can pull it. Preserve the failed task’s reason and network metadata before ECS removes its record. AWS lists pull-error branches and startup initialization failures.