Our incident: the OIDC subject that would not match

Symptom

Historical MiniTickets mismatch: the trust condition used repository names, but GitHub issued an immutable subject containing numeric ids. A representative STS denial is “Not authorized to perform sts:AssumeRoleWithWebIdentity”; the notes preserve no original error log.

Read the error first

STS refused the identity exchange: Allowed, specifically role trust. The caller had not reached ECR, ECS or Postgres. The human-readable repository name matching was insufficient because StringEquals compares the entire sub claim, the token’s statement of who is asking.

Decide

flowchart TB
  A[STS refuses exchange] --> B{Role and audience right?}
  B -->|No| C[Repair selection or audience]
  B -->|Yes| D{Subject includes numeric ids?}
  D -->|Yes| E[Compare immutable prefix]
  D -->|No| F[Check repository subject settings]
  E --> G{Branch or environment context?}
  G --> H[Use exact matching trust condition]

Likely causes

Ranked for this incident’s evidence:

1. The witnessed mismatch: old repo:maheshnd/minitickets prefix omitted owner and repository ids.
2. A branch subject was expected for a job with environment: dev; its context changes the suffix.
3. Another exact-match difference, such as wrong role, audience or branch; verify rather than guess.

How to check

Console

Repository → Actions → CD → run → publish/deploy job → Configure AWS credentials; identify the role and environment. IAM → Roles → that role → Trust relationships: compare its subject with repository OIDC configuration. The current prefix is repo:maheshnd@63289193/minitickets@1412723955.

CLI

export AWS_PROFILE=ticketlite
gh api repos/maheshnd/minitickets/actions/oidc/customization/sub
aws iam get-role --role-name minitickets-dev-gh-publish --query Role.AssumeRolePolicyDocument
aws iam get-role --role-name minitickets-dev-gh-deploy --query Role.AssumeRolePolicyDocument

Compare configuration and claims without logging the signed token or credentials.

CloudTrail

Search AssumeRoleWithWebIdentity at the job time. Inspect requested role, session name and error. A successful exchange exposes subject and audience fields; failed records may omit them. Record only those identity fields, never session credentials.

Fix

The code now builds trust from githubSubjectPrefix in infra/src/cicd.ts → createCicd. Publish appends :ref:refs/heads/main; deploy appends :environment:dev. Keep exact equality. Matching the real identity fixes this without granting other repositories access.

Prevent

Check the actual subject format when onboarding an account or changing job environments. GitHub explains immutable subjects: numeric owner and repository ids remain part of the repository segment. Document both job contexts in the OIDC chapter.

Say it in 30 seconds

Say it like this (20-30 seconds)

Our OIDC trust policy expected a name-only repository subject, but GitHub issued one with numeric owner and repository ids. STS correctly rejected the exact mismatch before returning credentials. We stored the real immutable prefix and built separate branch and environment subjects from it. I would inspect identity configuration before widening permissions, and preserve exact matching so a different repository cannot assume the role.

Advanced: What the historical record proves

The Step 12a record in frozen docs/31-oidc-publish.md states that name-only construction never matched this repository’s tokens and was replaced with githubSubjectPrefix. It does not preserve an original STS error, complete request trace or incident duration. Current code supplies the exact ids and policy. The AWS credentials action documents this representative denial. Repository recreation under the same name gets a different id; it should not inherit access. The environment subject checks environment identity; branch restrictions and required reviewers remain GitHub environment settings, not additional claims enforced by this IAM condition.