Real users get 429 Too Many Requests

Symptom

Diagnostic scenario: login or booking returns HTTP 429 with the API’s actual message, “Too many requests. Please try again shortly”. Requests include a Retry-After header.

Read the error first

MiniTickets’ rate limiter issued this application refusal. The request reached the API; it is not an IAM Allowed denial or a wrong Password. Identify the procedure: login limits by client IP; booking limits by session user id. A shared IP cannot merge different users’ booking buckets.

Decide

flowchart TB
  A[429 response] --> B{Which procedure?}
  B -->|Login| C{Client IP represents visitor?}
  C -->|No| D[Repair trusted proxy hops]
  C -->|Yes| E[Inspect shared IP and traffic]
  B -->|Booking| F[Inspect requests for same user]
  E --> G[Respect retry delay; review policy]
  F --> G

Likely causes

Ranked as checks for this scenario, not observed frequencies:

1. Login sees a proxy address for many visitors because too few proxy hops are trusted.
2. Real login users share an office or carrier public IP, or one client submits repeatedly.
3. Booking requests from one user exhaust their bucket, including duplicate clicks or aggressive retries.

How to check

Console

Browser Network: identify procedure, request rate and Retry-After. CloudWatch → /minitickets/dev/api: correlate request ids, status and logged remote address. ECS → api task definition: inspect TRUST_PROXY_HOPS. Committed ALB-only mode expects 1; CloudFront plus ALB expects 2.

CLI

export AWS_PROFILE=ticketlite
aws logs tail /minitickets/dev/api --since 10m
aws ecs describe-task-definition --task-definition minitickets-dev-api --query 'taskDefinition.containerDefinitions[?name==`api`].environment[] | [?name==`TRUST_PROXY_HOPS`]'

Compare with the task definition revision actually running. Do not dump all environment values or generate extra login traffic just to reproduce a rate limit.

CloudTrail

API 429s are application traffic, not CloudTrail management events. Correlate ECS RegisterTaskDefinition and UpdateService if the issue began after a proxy-setting change; use request logs for the rate-limit decision.

Fix

Correct identity extraction before changing capacity. Keep infra/src/entry/mode.ts → trustedProxyHops aligned with the actual entry path. For genuine shared-IP pressure, review the login policy and abuse trade-off using traffic evidence. Show the wait to the user.

Prevent

Retain proxy-hop tests and rate-limit tests. Login capacity is 20 with one token restored per 3 seconds; booking capacity is 10 with one per second. Booking’s retry rule rejects automatic 4xx retries. Do not increase trusted hops to address capacity.

Say it in 30 seconds

Say it like this (20-30 seconds)

I first identify which procedure returned 429 and which identity its bucket uses. MiniTickets limits login by IP and booking by user, so a proxy mistake affects them differently. I compare logged client addresses with the actual proxy path, then check shared-IP traffic and repeated requests. I repair identity extraction before tuning capacity, preserve abuse protection, and tell clients when they can retry.

Advanced: Refusal modes and privacy

apps/api/src/services/rate-limits.ts → RATE_POLICIES selects capacity. apps/api/src/queries/rate-limits.ts → takeToken uses one atomic Redis script and hashes identities in key names. A Redis failure becomes 503, not 429: protection fails closed. Requests with a session can fail during session lookup even earlier. Trusting too many hops accepts client-supplied forwarding data and can let callers choose their apparent IP. A localhost Service Connect caller is deliberately never trusted as an external proxy.