A target stays unhealthy

Symptom

Diagnostic scenario: EC2 → Target groups shows Request timed out or Health checks failed with these codes: [404]. ECS may show a running container, then replace it after failed health checks.

Read the error first

The ALB checks the target’s private IP and port. A timeout suggests Reach or a slow process. A 404 proves the process replied, so check the path. This is not an IAM Allowed denial or a DB Password failure. Container health is a separate check from target health.

Decide

flowchart TB
  A[Unhealthy target reason] --> B{HTTP status returned?}
  B -->|404 or redirect| C[Check path and matcher]
  B -->|5xx| D[Read health-route logs]
  B -->|timeout| E{Process listening correctly?}
  E -->|no| F[Check bind address and port]
  E -->|yes| G[Check SG path and latency]

Likely causes

Rank within the observed branch:

1. Wrong status: health path changed. API must check /health on 4000; web checks / on 3000; both expect 200.
2. Timeout after network change: task security group no longer allows ALB’s security group on that port.
3. Timeout after image change: process binds only to localhost, uses the wrong port, crashes, or takes too long.
4. Only during startup: startup exceeds the configured grace period. Establish that before extending it.

How to check

Console

EC2 → Target groups → Targets → Health status details, then Health checks. ECS → task → Health status, logs and Network. Compare API HOST=0.0.0.0 or web HOSTNAME=0.0.0.0 with task port mappings and SG rules.

CLI

Replace target-group ARN and security-group ID with those attached to the unhealthy target.

export AWS_PROFILE=ticketlite
aws elbv2 describe-target-health --target-group-arn <target-group-arn>
aws elbv2 describe-target-groups --target-group-arns <target-group-arn> --query 'TargetGroups[].{Port:Port,Path:HealthCheckPath,Matcher:Matcher,Timeout:HealthCheckTimeoutSeconds}'
aws ec2 describe-security-groups --group-ids <task-security-group-id> --query 'SecurityGroups[].IpPermissions'

CloudTrail

Search ModifyTargetGroup, UpdateService, RegisterTaskDefinition and security-group ingress changes around the first failure. CloudTrail records changes, not individual probes; the target reason and application logs explain those probes.

Fix

Restore the correct path, bind address or SG source. API inbound TCP 4000 and web inbound TCP 3000 allow the ALB group in infra/src/security.ts → createSecurityGroups. Keep ALB health independent of temporary DB failures: use /health, not /ready.

Prevent

Review target and container checks together. Target checks have a 15-second interval, 5-second timeout and 200 matcher; ECS allows 60 seconds for startup. Test that the production image listens on the container interface before deploying it.

Say it in 30 seconds

Say it like this (20-30 seconds)

Running is not the same as healthy. I read the target health reason first: a wrong HTTP code means the ALB reached an application, while a timeout could mean blocked traffic, a wrong bind address or a slow process. I compare the actual target port and health path with the task and security group. I fix that mismatch, keep dependency failures out of liveness, and verify the target becomes healthy without repeated replacement.

Advanced: Read the two health systems separately

A container health check runs inside the task. The API’s command fetches localhost:4000/health; it cannot prove the ALB can reach the task interface. The ALB probes from outside the task. Both settings live in infra/src/compute/task.ts → createApiTask and infra/src/entry/target-groups.ts → createTargetGroups.

The group requires three failures before an established target becomes unhealthy and two successes to recover. Newly registered targets have different initial-health behavior: AWS needs one successful initial check. Do not estimate exact recovery time by multiplying thresholds alone; startup, registration and probe timing also matter. AWS defines health states and reasons.

If all targets are unhealthy in every enabled Availability Zone, ALB can fail open and route to them. A green container or an occasional 200 therefore does not replace target-health review.