A scheduled Lambda gets AccessDenied after weeks

Symptom

Interview scenario, explain-only: a scheduled Lambda reads S3 objects and writes to Postgres. After weeks it logs AccessDenied. MiniTickets has no deployed scheduled S3-to-DB Lambda; its ticket Lambda provides real examples of scoped permissions.

Read the error first

Find the failing operation: S3 GetObject, KMS Decrypt, or Secrets Manager GetSecretValue. These are Allowed failures. A connection timeout suggests Reach; pool or query timeouts can also mean saturation. Postgres password authentication failed is Password. IAM means AWS permissions, not database login. In AWS errors, no identity-based policy allows points to a missing grant; explicit deny points to a refusal that an extra allow cannot override.

Decide

flowchart TB
  A[First failed operation] --> B{AWS AccessDenied?}
  B -->|yes| C[Check action, resource and caller]
  B -->|no| D{Database replied?}
  D -->|password error| E[Refresh cached DB secret]
  D -->|timeout| F[Check network and DB saturation]
  C --> G[Compare changes since last success]

Likely causes

For this scenario, rank by the evidence around the first failure:

1. New objects only: a new prefix falls outside the policy, or those objects use a different KMS key.
2. All objects: role/bucket/key policy changed; an organization service control policy (SCP), role permissions boundary or VPC endpoint policy now limits access.
3. One uploader only: object ownership differs where access control lists (ACLs) remain enabled.
4. S3 succeeds, DB fails: a rotated DB secret stayed cached. Rotated long-lived AWS access keys matter only if someone configured them; Lambda normally uses temporary execution-role credentials.

How to check

Console

Lambda → Functions → function → Configuration → Permissions: identify the execution role. S3 → General purpose buckets → bucket → Objects → failed object → Object overview: check its key and encryption. Bucket → Permissions → Object Ownership; Secrets Manager → Secrets → secret → Rotation configuration. Inspect metadata, never reveal secret values.

CLI

Set the local profile; replace placeholders with the scenario’s function, bucket and secret ARN.

export AWS_PROFILE=ticketlite
aws lambda get-function-configuration --function-name <function-name> --query '{Role:Role,Vpc:VpcConfig}'
aws s3api get-bucket-ownership-controls --bucket <bucket-name>
aws secretsmanager describe-secret --secret-id <secret-arn> --query '{Changed:LastChangedDate,Rotated:LastRotatedDate}'

CloudTrail

Compare caller ARN, action, resource and denial time. Search GetSecretValue and KMS Decrypt; S3 GetObject needs previously enabled data-event logging, not Event history. Check policy-change events around the last success.

Fix

Correct the exact resource grant or reviewed deny; grant KMS decryption only when required. Refresh a stale DB password and reconnect. A hypothetical DB-writing Lambda also needs a VPC network path and database grants; IAM alone supplies neither.

Prevent

Record action/resource/error category without secrets. Test new prefixes, encryption keys and rotation. Compare MiniTickets’ real prefix grants in infra/src/tickets/iam.ts → createTicketRole.

Say it in 30 seconds

Say it like this (20-30 seconds)

I would locate the first failed operation before changing permissions. S3 or KMS AccessDenied means authorization; a database connection timeout needs networking and saturation checks; a password error means authentication. For a previously working job, I compare policy changes, object prefixes, encryption keys and upload ownership with the last success. If S3 succeeded, I check cached database credentials after rotation. I fix the narrow cause and add a rotation or new-object regression check.

Advanced: Different changes need different repairs

A permissions boundary limits what a role can receive; an SCP limits permissions in an organization. Neither grants permission by itself. A new broad role grant cannot override an explicit deny. Read the named policy type before asking its owner for a change. AWS explains denial messages.

An S3 object encrypted with SSE-KMS needs permission to decrypt its actual KMS key, including the key’s policy. Inspect a failed object’s encryption metadata rather than assuming the bucket’s current default describes older objects. Changing a bucket’s default encryption does not rewrite existing objects. AWS lists SSE-KMS permissions.

With BucketOwnerEnforced, ACLs are disabled and the bucket owner owns objects. With ACLs enabled, an external uploader can own an object. Check that setting before blaming an ACL. AWS documents Object Ownership. MiniTickets explicitly uses BucketOwnerEnforced and SSE-S3, not SSE-KMS, in infra/src/tickets/storage.ts → createTicketStorage.

Do not confuse a rotated AWS access key with a rotated Postgres password. MiniTickets’ ECS API receives its DB password only at task start. New tasks obtain the current value; old tasks do not refresh it automatically. That is an existing limitation, not evidence of a scheduled Lambda incident. See infra/src/compute/task.ts → createApiTask. S3 data events require logging configuration.