Our incident: deleting a Lambda trigger needs read permission too

Symptom

CD run 39181968474 failed while turning the environment off. The deploy role minitickets-dev-gh-deploy could request deletion, but Pulumi’s waiter failed with “not authorized to perform: lambda:GetEventSourceMapping on resource: *” and “no identity-based policy allows”. No successful recovery is recorded here.

Read the error first

Lambda refused Allowed access for the deploy role’s GetEventSourceMapping call. An event source mapping connects a queue to a function. Pulumi’s waiter repeatedly reads its state after requesting deletion. The crucial detail is resource *: an Allow on mapping ARNs does not cover that authorization check. This is neither Reach nor Password; changing the worker’s execution role would address another caller.

Decide

flowchart TB
  A[GetEventSourceMapping denied] --> B{Resource in error?}
  B -->|Wildcard| C{Get allowed on wildcard?}
  B -->|Mapping ARN| D[Check ARN scope and conditions]
  C -->|No| E[Add only Get on wildcard]
  C -->|Yes| F[Inspect effective policy limits]
  D --> G[Compare live policy and source]
  E --> H[Test waiter read and regional deny]

Likely causes

Investigation order for this error, not measured frequency:

1. ARN-only mapping access misses the waiter’s GetEventSourceMapping check on *, as this run shows.
2. The live policy or approved source differs from the policy being inspected; verify both before retrying.
3. The caller, Region or policy conditions make a present grant inapplicable.

How to check

Console

GitHub → failed compute-off step: preserve the principal, action, resource, time and source SHA. IAM → minitickets-dev-gh-deploy → inline policy deploy: compare its actions and resources. Lambda → function → Configuration → Triggers: inspect any remaining mapping.

CLI

For the last command, set MAPPING_UUID to the UUID from the mapping’s Pulumi diagnostic or listing.

export AWS_PROFILE=ticketlite
aws iam get-role-policy --role-name minitickets-dev-gh-deploy --policy-name deploy --query PolicyDocument
aws lambda list-event-source-mappings --region us-east-1 --query 'EventSourceMappings[].{UUID:UUID,Function:FunctionArn,State:State}'
aws cloudtrail lookup-events --region us-east-1 --lookup-attributes AttributeKey=EventName,AttributeValue=GetEventSourceMapping --query 'Events[].{Time:EventTime,User:Username,Id:EventId}'
aws lambda get-event-source-mapping --region us-east-1 --uuid "$MAPPING_UUID"

Preserve the original UUID: listing may no longer return a mapping whose deletion started.

CloudTrail

Correlate DeleteEventSourceMapping and GetEventSourceMapping by UUID, role session and time; inspect errorCode and errorMessage. Check PutRolePolicy history for policy changes. Compare those events with the recorded CD failure; no correlated CloudTrail trace is recorded here.

Fix

Make create, get, list, update and delete permissions explicit in infra/src/cicd.ts → createCicd. Keep update/delete and tag operations scoped to this account and Region’s mapping ARNs. Get needs a separate Resource: "*" Allow for this waiter; create/list also use *. Keep the regional Deny. Deliver the reviewed repair through the owner before retrying CD.

Prevent

infra/src/cicd.test.ts captures the real policy through Pulumi mocks and checks lifecycle permissions and scopes, including the wildcard waiter read and regional Deny, without AWS. Keep tag operations scoped. Review the other managed resources’ read/describe/list calls too. Inspect the deploy role’s permissions →.

Say it in 30 seconds

Say it like this (20-30 seconds)

Our compute-off deployment requested deletion of a Lambda trigger, then its waiter failed on GetEventSourceMapping against resource star. The existing mapping ARN grant did not cover that check. We give only this read a separate wildcard Allow, retain the regional Deny, and keep updates, deletes and tags ARN-scoped. Unit tests cover the lifecycle and waiter scope. The logs establish the failure; they do not establish successful recovery.

Advanced: Source, deployment and deletion are separate evidence

The failed run used source e01ae485acdc5290d6db3ac04800ee95dc043198. On 2026-10-19, worker mapping deletion started at 00:53:23 UTC and failed at 00:53:37 UTC; the tickets mapping failed at 00:53:47 UTC. Both failures named GetEventSourceMapping on *.

The prior source already granted lambda:* on this account and Region’s mapping ARNs in JobPipeline. That grant could not cover the wildcard resource named by these errors. The deletion-time read explains why adding another ARN-scoped Get would be insufficient; the logs do not establish Lambda’s internal authorization sequence or the precise moment the mappings disappeared. No successful rerun or complete live-policy check is recorded here.

The mapping scope is arn:aws:lambda:us-east-1:258237553751:event-source-mapping:*. AWS’s Lambda authorization table supports mapping ARN scope for get/update/delete and tags. The recorded waiter denial nevertheless names *, so Get must also cover that resource. This permits reads across this account’s mappings in us-east-1; the regional Deny still applies. Create/list have no mapping ARN scope. Mapping UUIDs have no minitickets-dev- prefix.

Deletion is asynchronous. GetEventSourceMapping reads the mapping by UUID. A later ResourceNotFound response establishes absence at that read; it does not prove that the earlier Pulumi run finished or reconciled its state.