A task cannot read its secret

Symptom

Diagnostic scenario: a stopped task reports ResourceInitializationError, with AccessDeniedException and secretsmanager:GetSecretValue in the detail. The application has not started. No historical MiniTickets outage is asserted here.

Read the error first

ECS asks Secrets Manager for startup values using the execution role. This denial is Allowed, not a database Password failure. A timeout contacting Secrets Manager is Reach. Read the full suffix: a missing identity-policy grant differs from an explicit deny, and ResourceNotFoundException points toward a secret/version reference.

Decide

flowchart TB
  A[Secret initialization failed] --> B{Timeout?}
  B -->|yes| C[Check task network]
  B -->|no| D{Access denied?}
  D -->|yes| E[Check execution-role policy and deny]
  D -->|no| F[Check ARN, JSON key and current version]
  E --> G[Check KMS only for a customer key]

Likely causes

Rank by the error, then recent changes:

1. Denied after recreation: read-secrets still names an old secret ARN, or the task uses the wrong execution role.
2. Not found/key error: wrong region, ARN, JSON key, or missing current secret version.
3. Timeout: task cannot reach Secrets Manager’s HTTPS endpoint.
4. Customer-managed KMS key only: missing decryption permission or restrictive key policy. MiniTickets uses the default Secrets Manager key for these secrets.

How to check

Console

ECS → stopped task → Stopped reason → task definition → Task execution role and secret references. Compare IAM → read-secrets with Secrets Manager → secret ARN and Versions. Do not reveal the secret value.

CLI

Replace the task-definition ARN and secret ARN with the failing task’s references.

export AWS_PROFILE=ticketlite
aws ecs describe-task-definition --task-definition <task-definition-arn> --query 'taskDefinition.{ExecutionRole:executionRoleArn,Secrets:containerDefinitions[].secrets}'
aws iam get-role-policy --role-name minitickets-dev-api-execution-role --policy-name read-secrets
aws secretsmanager describe-secret --secret-id <secret-arn> --query '{ARN:ARN,Key:KmsKeyId,Versions:VersionIdsToStages}'

CloudTrail

Search GetSecretValue at startup. Check the execution-role identity, secret identifier and error fields without copying response data. Inspect PutRolePolicy or key-policy changes around the first failure. A network timeout may leave no Secrets Manager event.

Fix

Correct the reference or the smallest execution-role grant in infra/src/compute/iam.ts → createIamRoles. The API requests username and password JSON keys in infra/src/compute/task.ts → createApiTask. Use the web’s separate execution role for its revalidate secret.

Prevent

Pass secret ARNs from their defining resources instead of copying strings. Start tasks only after a secret version exists. After rotation, arrange a reviewed rollout: existing environment variables do not refresh themselves.

Say it in 30 seconds

Say it like this (20-30 seconds)

An ECS startup secret failure belongs to the execution role, because ECS retrieves the secret before the application starts. I read the stopped reason to separate access denied, a missing secret or JSON key, and a network timeout. Then I compare the task’s exact secret ARN with the role policy and current version. I repair only that reference or permission and make sure rotation launches tasks that receive the new value.

Advanced: Startup success does not guarantee a current password

API startup reads the RDS-managed JSON secret, the Redis URL, and the revalidate secret when entry is enabled. Web startup reads only the revalidate secret. Its role is defined in infra/src/compute/web-iam.ts → createWebIamRole.

The API task’s password can become stale after RDS rotates it. A running container then gets a Postgres authentication error, not a startup IAM denial. See the scheduled-Lambda diagnosis. AWS documents startup injection and rotation. AWS lists initialization error branches.