The API cannot reach Postgres

Symptom

Diagnostic scenario: API logs contain ETIMEDOUT, /ready fails, or logs instead say password authentication failed. Those messages describe different failure stages; no historical outage is claimed here.

Read the error first

DNS or TCP timeout is Reach: can the API find and connect to RDS, AWS’s managed database? A certificate error concerns TLS, which encrypts and verifies the connection. Postgres authentication is Password; database permission denied is a SQL grant issue. MiniTickets uses password login, so IAM Allowed does not grant SQL access.

Decide

flowchart TB
  A[First database error] --> B{DNS resolves?}
  B -->|no| C[Check endpoint and VPC DNS]
  B -->|yes| D{TCP connects?}
  D -->|no| E[Check DB state, SGs and routes]
  D -->|yes| F{TLS succeeds?}
  F -->|no| G[Check CA and hostname]
  F -->|yes| H[Check password or SQL grants]

Likely causes

Rank within the error branch:

1. Timeout after network change: DB inbound TCP 5432 no longer names API’s security group, or the task has a different group.
2. Name/endpoint failure: stale DB_HOST after recreation or VPC DNS changed.
3. Authentication after rotation: the running task still holds the old injected password.
4. TLS error: missing RDS certificate authority (CA), wrong hostname or client SSL configuration. Also check RDS availability before blaming the client.

How to check

Console

RDS → minitickets-dev-db → status, endpoint, port, VPC and SG. ECS → API task → Network: confirm api-sg. EC2 → db-sg → Inbound: TCP 5432 from api-sg only. Compare API error timestamps with RDS events and secret rotation metadata.

CLI

Replace the group ID with db-sg’s ID. These commands reveal metadata, never the password.

export AWS_PROFILE=ticketlite
aws rds describe-db-instances --db-instance-identifier minitickets-dev-db --query 'DBInstances[].{Status:DBInstanceStatus,Endpoint:Endpoint,Groups:VpcSecurityGroups,Public:PubliclyAccessible}'
aws ec2 describe-security-groups --group-ids <db-security-group-id> --query 'SecurityGroups[].IpPermissions'

CloudTrail

Check ModifyDBInstance and security-group ingress changes at the first failure. CloudTrail does not record ordinary Postgres connections or SQL. Use API/RDS logs for password and query errors; use Secrets Manager rotation metadata for credential timing.

Fix

Restore only API-to-DB TCP 5432 in infra/src/security.ts → createSecurityGroups. Refresh tasks after credential rotation. Repair TLS trust through apps/api/src/config/database.ts → DatabaseEnvSchema; keep certificate verification enabled. Do not make RDS public to repair an internal path.

Prevent

Keep endpoint and SG references wired from resources. Compare /health with /ready: liveness can pass while DB readiness fails. Test production TLS configuration and plan credential refresh for long-running tasks.

Say it in 30 seconds

Say it like this (20-30 seconds)

I classify the database error by stage: DNS, TCP reachability, TLS, then authentication and SQL permissions. In MiniTickets, RDS is private and inbound 5432 allows only api-sg, so I compare the actual task group and database endpoint before touching credentials. A password error proves the server replied and may follow rotation of a startup-injected secret. I repair that stage, keep TLS verification enabled, and check readiness without making liveness depend on the database.

Advanced: A private DB does not need a NAT gateway for this path

API and RDS communicate through the VPC’s local route. The database has no public IP and lives in private subnets. It does not need internet access to answer API queries. Security groups allow replies to established traffic, so the DB group’s empty egress list does not block query responses.

The API pool has a 2-second acquisition/connect timeout, a 5-second statement timeout and at most 10 connections in apps/api/src/db/client.ts → pool. A pool-acquisition timeout can also mean all connections are busy; inspect connection and query load before treating every timeout as dropped network packets.

RDS setup is in infra/src/data/postgres.ts → createPostgres; the task reads DB credentials only at startup in infra/src/compute/task.ts → createApiTask. AWS documents startup-injected secrets.