AccessDenied right after a policy change: propagation delay, or did the policy really change?

Symptom

Owner-reported P1 evidence: publish for a6c3993 failed its image-existence check with “because no identity-based policy allows the ecr:BatchGetImage action”. A later live policy read included the worker repository and action. No timing or retry outcome was recorded; propagation was a guess, not a proven incident.

Read the error first

ECR refused Allowed access: the publish role requested ecr:BatchGetImage on minitickets-dev-worker. “No identity-based policy allows” points to a missing applicable Allow; an explicit deny is a different clue. This is neither Reach nor Password. Read the principal, action and resource before changing anything.

Decide

flowchart TB
  A[Missing Allow error] --> B{Exact grant in live policy?}
  B -->|No| C[Find missing or reverted grant]
  B -->|Yes| D{Policy changed recently?}
  D -->|Yes| E[Compare event times; bounded retry]
  D -->|No| F[Inspect effective policy conditions]
  E -->|Same denial repeats| C
  C --> G[Compare stack history and PutRolePolicy]

Likely causes

Investigation order, not measured frequency:

1. The live policy lacks the repository grant, including after an older deploy reverted it.
2. The request preceded the update. A later correct policy does not establish what existed earlier.
3. A fresh correct update had not propagated. IAM distributes changes asynchronously, but this explanation needs evidence.

How to check

Console

GitHub → failed publish → image check: capture principal, action, resource and time. IAM → minitickets-dev-gh-publish → push-images: inspect the live grant. Compare Pulumi update history and the approved source SHA; correct local code does not prove the live policy stayed correct.

CLI

export AWS_PROFILE=ticketlite
aws iam get-role-policy --role-name minitickets-dev-gh-publish --policy-name push-images --query PolicyDocument
aws cloudtrail lookup-events --region us-east-1 --lookup-attributes AttributeKey=EventName,AttributeValue=PutRolePolicy --query 'Events[].{Time:EventTime,User:Username,Id:EventId}'

CloudTrail

Inspect PutRolePolicy: event time, caller, role name and policy document. Compare the repository ARN with the denied resource. This can reveal an older deploy restoring an older policy; a later successful read cannot prove the earlier authorization state.

Fix

Repair the live grant through reviewed infrastructure. A bounded retry is reasonable after a fresh correct update; repeated denial requires checking for a revert. Keep repository grants aligned in infra/src/publish-policy.ts → publishPolicyDocument.

Prevent

Reject stale infrastructure with scripts/deploy-guard.sh. Check live IAM and deployment history before blaming propagation. The proven recurrence is the stale-approval incident, where old source removed the grant.

Say it in 30 seconds

Say it like this (20-30 seconds)

I read the denied principal, action and resource, then inspect the live policy and its change history. Our first worker-publish failure was not proven to be propagation. The same error later came from an old approved deploy reverting the policy. We repaired that grant and rejected stale infrastructure runs. A bounded retry can suit a fresh change, but repeated denial means checking whether the policy changed again.

Advanced: The two failures, with the evidence kept separate

The owner supplied the first error for CD publish a6c3993:

User: arn:aws:sts::258237553751:assumed-role/minitickets-dev-gh-publish/gh-publish-38783979040 is not authorized to perform: ecr:BatchGetImage on resource: arn:aws:ecr:us-east-1:258237553751:repository/minitickets-dev-worker because no identity-based policy allows the ecr:BatchGetImage action

A full local Pulumi update had added the worker repository to the publish policy. A subsequent live get-role-policy read showed the repository and action present. The request might have preceded that update, or propagation might have delayed visibility. No recorded timeline or retry result settles those alternatives. An earlier Step 12a propagation precaution was not evidence of an incident either; that login failure involved the immutable OIDC subject.

The same denial for 95e4354 had a proven different cause: approving stale 6ea9575 applied old infrastructure, reverted the publish policy and removed the worker repository grant, while deleting ten newer resources. The owner restored the worker repository and publish/deploy policies with a targeted apply, waited about one minute, then reran the failed job successfully. That minute is an observed recovery interval, not an IAM propagation guarantee.

infra/src/cicd.ts → createCicd attaches the publish policy. AWS documents IAM’s eventual consistency: accepted changes can become visible at different times. This explains a possibility; it does not retrospectively establish the cause of the first failure.