The load balancer returns 502 or 503

Symptom

Diagnostic scenario: pages return 503 Service Temporarily Unavailable, or requests intermittently return 502 Bad Gateway. First establish that the Application Load Balancer (ALB) generated the error; an application can also return a 5xx.

Read the error first

An ALB-generated 503 usually means no registered targets, or all registered targets are unused. An ALB-generated 502 means a connection or response failed, such as a reset or malformed HTTP. This starts with Reach and HTTP behavior, not AWS Allowed or DB Password. AWS lists the status-specific causes.

Decide

flowchart TB
  A[Browser gets 5xx] --> B{Who generated it?}
  B -->|target| C[Read application logs]
  B -->|ALB| D{Status?}
  D -->|503| E[Inspect registered and unused targets]
  D -->|502| F[Inspect resets and invalid responses]
  D -->|504| G[Inspect reachability and response time]

Likely causes

Rank inside the observed branch:

1. 503: selected target group is empty after task startup failures, or targets are unused because the listener/AZ configuration does not use them.
2. 502 during replacement: a task resets or closes a request; compare Spot interruption, crash and deployment times.
3. 502 without replacement: invalid response headers or incompatible connection lifetime.
4. 504 instead: blocked ALB-to-task traffic or a slow target. Unhealthy targets alone do not prove 503; ALB can fail open when all are unhealthy.

How to check

Console

EC2 → Load Balancers → minitickets-dev → Resource map, Listeners and rules. /trpc/* goes to API; other paths go to web. Compare target states, ECS service events, and CloudWatch’s ELB versus Target 5xx metrics.

CLI

Replace the target-group ARN with the group selected by the failing path.

export AWS_PROFILE=ticketlite
aws elbv2 describe-target-health --target-group-arn <target-group-arn>
aws ecs describe-services --cluster minitickets-dev --services minitickets-dev-api minitickets-dev-web --query 'services[].{Name:serviceName,Running:runningCount,Desired:desiredCount,Events:events[:5]}'

CloudTrail

Search UpdateService, ModifyListener, ModifyRule and target registration changes around the failure. CloudTrail explains configuration changes, not each HTTP request. Use metrics and application logs; ALB access logs help only if already enabled.

Fix

Restore the selected group’s running tasks or listener/AZ configuration. For 502, repair the crash, response or connection handling supported by evidence. For health failures, use the target case. MiniTickets uses one Spot task per service, so interruption can remove its only serving process.

Prevent

The alb-5xx alarm combines ALB and target errors. CD checks running image identity as well as stability. Review replacement behavior and the 30-second draining setting in infra/src/entry/target-groups.ts → createTargetGroups.

Say it in 30 seconds

Say it like this (20-30 seconds)

I first separate an ALB error from an application error using the ELB and target metrics. For an ALB 503, I inspect the selected target group for absent or unused targets and trace back to ECS startup or listener configuration. For 502, I look for connection resets, task replacement and invalid responses. I correlate timestamps before changing anything, then verify the intended tasks serve requests and the error metric recovers.

Advanced: All unhealthy is not the same as no targets

When every target is unhealthy in all enabled Availability Zones, ALB can route to those unhealthy targets: fail-open behavior. That can produce successful responses or other errors, depending on the underlying failure. It is unsafe to infer 503 from HealthyHostCount = 0 alone. AWS documents fail open.

In committed ALB-only mode, HTTPS defaults to web and priority 100 sends /trpc/* to API: infra/src/entry/https-listener.ts → createHttpsListener. CloudFront mode is off. If later enabled, a CloudFront 504 can mean its TCP 80 path to ALB is blocked; missing origin-header authorization instead yields the listener’s fixed 403. These are separate from an ALB-generated 502 or 503.