Design decisions

Each decision says what we chose, why, and what it costs. In an interview, the trade-off is the part that shows judgement. Decisions are added as chapters are written; Parts 0 to 6 are covered so far.

The AWS foundations

Decision	Why	Trade-off	Chapter
One account, region us-east-1, names from the stack	Simple; a second stack gets its own names with no code change	Dev and prod in one account would share a blast radius	Account and region
No NAT gateway; tasks in public subnets with public IPs	Saves about 30+ USD a month for one dev environment	Tasks have public IPs; security groups do all the protecting	Route tables
Data in private subnets with only a local route	No network path from the internet to Postgres or Valkey	The databases cannot call out (they never need to)	VPC and subnets
Security groups reference each other, not CIDRs	Allows exactly the tasks that use a group; survives changing IPs	A wrong reference fails silently as a timeout	Security groups
Separate execution and task roles; web has no task role	The code never holds the power to read secrets	More roles to manage	IAM roles
RDS manages the master password	The password never exists in code, config or Pulumi state	Running tasks keep the old value after a rotation	Secrets
Single-AZ RDS, 1 day of backups, no final snapshot	Cheap dev environment	Switching data off deletes the data	RDS Postgres
One Valkey node, no backups	Everything in it is short-lived or rebuildable	Losing it logs everyone out	ElastiCache Valkey
Immutable image tags (the git SHA), built once for ARM64	Every running task maps to one commit; ARM is cheaper	Images must match the ARM64 platform	ECR and images
One task each on Fargate Spot, circuit breaker on	About 70% cheaper; failed deploys roll back	A Spot reclaim means a short outage	ECS on Fargate
Service Connect for web ↔ api calls	Stays inside the VPC; works in CloudFront mode	Two proxy ports and two extra rules	ECS on Fargate
ALB health check on /health, not /ready	A database blip does not take every task out	The ALB keeps tasks that cannot reach Postgres	Load balancer
DNS at Cloudflare, records added by hand	The domain was already there	Manual step; a new ALB needs a CNAME update	DNS and certificates
CloudFront built but off (ALB-only mode)	The new account needed CloudFront verification	No edge caching; the ALB serves HTTPS itself	CloudFront
Two locks for CloudFront: prefix list + secret header	Other people’s distributions share CloudFront’s IPs	One more secret to keep in step	CloudFront

The app inside

Decision	Why	Trade-off	Chapter
Procedures → services → queries / queues	Each rule lives in one place; services test without a database	More files; booking checks break the rule on purpose	Layers
Server-side sessions in Valkey, not JWTs	Instant revocation, including log out everywhere	One Valkey lookup per request	Sessions
Role read from Postgres on each admin request	Granting or removing admin counts at once	One small query per admin request	Roles in the app
Services throw kinds; one mapper makes HTTP codes	Services know no HTTP; internals never leak	A plain Error becomes a 500	Errors
Three cache layers, cleared on write	Fast reads; changes show in milliseconds	Every booking clears every cached list	Caching
SELECT … FOR UPDATE plus a CHECK constraint	No oversell; clear 404 / 400 / 409 answers	A hot row on very popular events	Row locks
Idempotency key in the booking row	Commits with the booking; can never disagree	Keys are kept forever (small)	Idempotency
Keyset pagination and a trigram index	Deep pages stay cheap; contains-search is indexed	No jumping to page N	Search and pagination
Valkey rate limits (fail closed) and seat gate (fail open)	Protects Postgres on sale day; Postgres stays the authority	The counter can drift and needs repair	Rate limits
Waiting room, seat holds, autoscaling: explained, not built	Not needed at our traffic	A very big sale would need them	Rate limits
Nonce CSP, Origin check on mutations, HSTS for 1 day	Blocks injected scripts and CSRF; safe start for HSTS	HTML cannot be shared-cached	Security headers

Async and serverless

Decision	Why	Trade-off	Chapter
Transactional outbox, relay inside the api	The booking and its message commit together; no extra service	At-least-once: a crash can republish a message	The outbox
SNS topic fanning out to one SQS queue per consumer	The api publishes once; email and tickets fail and retry apart	Two more queues and policies to run	SNS fan-out
maxReceiveCount 3, backoff 10 / 40 / 120 s, DLQ kept 14 days	Bad messages stop retrying; a person sees each one	A long outage sends healthy messages to the DLQ	SQS and the DLQ
Lambda outside the VPC, partial batch failures, maximumConcurrency	No NAT; only failed messages retry; pollers never over-fetch	It cannot reach Postgres, so messages carry all they need	The Lambda worker
Idempotent worker in DynamoDB, keyed by booking, send then record	One email per booking; a crash never loses one	A rare duplicate email in the crash window	DynamoDB
Private bucket, presigned URLs for 120 s	Files never pass through the api; the bucket stays private	A new link for every page view	S3 and presigned URLs
SES with Easy DKIM, still in the sandbox	Signed mail; no production review needed to learn	Only verified recipients receive email	Email with SES

Delivery

Decision	Why	Trade-off	Chapter
CD starts on workflow_run after CI is green on main	Only tested code can ship	CD waits for the slowest CI job	CI
Build once for ARM64, promote the same bytes	What was scanned is what runs	Rollback targets must be kept in ECR	Images and scans
OIDC with exact sub checks; separate publish and deploy roles	No stored keys; the code-installing job cannot deploy	The deploy role is still broad	OIDC
Migration as a one-off Fargate task between two phases	A failed migration leaves the running app untouched	Migrations must stay compatible with the old code	Migrations
Image tag in SSM, switches in the committed config	Neither a laptop nor CD rolls back by accident	Two places to look for “what runs”	Two-phase deploy
Deploy guard against the last applied commit	An old approved run can no longer delete newer resources	A deliberate infra rollback needs a manual flag	Rollback and the guard
A bootstrap script for the 14 resources CD needs to start	CD cannot create the roles it logs in with	One manual, owner-run step per new account	Bootstrap

Operating it

Decision	Why	Trade-off	Chapter
JSON logs with a correlation id carried into messages	One query follows a booking across the api and the worker	Logs kept only 7 days	JSON logs
Four alarms on customer impact and stuck work	Few alarms, each worth waking for	Slow-but-working problems raise no alarm	Alarms
One dashboard in code, parts follow the switches	Free, reviewed, never drifts	No database metrics on it	The dashboard
Billed layers behind committed switches; a warning-only budget	Pay only while working; changes are reviewed	Switching data off deletes the data	Cost switches
Runbooks with read-only checks first and no pasteable write commands	Calm, repeatable first minutes	Writing and updating them takes time	Runbooks