Why can rotating a secret break only some running processes?

Say it in 30 seconds

Say it like this (20-30 seconds)

Secret rotation changes stored credentials, not every running consumer at once. MiniTickets injects database credentials when ECS starts a task, while the ticket worker caches a successfully fetched signing secret per process. Old and new processes can therefore use different values. I first identify the consumer and refresh behavior, then plan a compatible rollout. Signing-key rotation also needs old-ticket verification support; restarting workers alone does not solve that lifecycle problem.

The deeper answer

There are three separate states: the value in Secrets Manager, the credential accepted by the destination and the value held by each consumer. A mismatch can appear only when a database connection is renewed, making an incident look intermittent.

AWS documents that ECS environment secrets do not update automatically. New tasks retrieve current values. A controlled replacement must preserve serving capacity and verify fresh connections; reading or logging the secret is not a useful diagnostic requirement.

The ticket signingSecret helper caches only successful reads. A transient failed read can be retried, but a successful cached value has no refresh interval. The current token format has no key identifier or multi-key rotation protocol. A future rotation design would retain verification keys for valid issued tickets and define revocation and expiry rules. Do not describe that design as implemented merely because Secrets Manager stores the secret.

Counter-questions

Would adding IAM permissions fix password authentication failures?

Not if retrieval succeeded and the consumer holds an old password. Separate authorization to fetch a secret from authentication to the destination.

Why cache a secret at all?

It avoids a network call on every job. The trade-off is delayed rotation visibility and the need for an explicit refresh policy.

Can you immediately delete the old ticket key?

That can invalidate already issued tickets. Define the validity period and verifier behavior before replacing signing material.

Where in MiniTickets

Read the path chapter, select the relevant map piece →, then inspect apps/worker/src/tickets/assets.ts → signingSecret. Work through the related case for a diagnostic example.