JSON logs and correlation IDs

In 1 minute

A booking touches the api, an SNS topic, a queue and a Lambda, each with its own logs. To follow it, every request gets a correlation id, a UUID written on every log line of that request. The api copies it into the queue message, so the worker’s lines carry it too. All lines are JSON, so CloudWatch Logs Insights can filter on any field: one query shows the whole booking.

Picture it

flowchart TB
  B["browser"] -->|"x-correlation-id (optional)"| A["api: request.id = UUID<br/>every line: correlationId"]
  A -->|"message.correlationId"| Q["SNS → SQS"]
  Q --> W["worker: every line:<br/>correlationId, sqsMessageId, tries"]
  A --> L[("CloudWatch Logs")]
  W --> L

How it works

1. Pick the id. If the caller sends a valid UUID in x-correlation-id, the api keeps it; anything else gets a new random UUID. A strict format keeps fake text out of our logs.
2. Write it everywhere. Fastify keeps it as the request id and writes it on every line of that request as correlationId. Every answer, errors too, sends it back in the same header.
3. Carry it on. The booking service puts it in the “booking confirmed” message.
4. The worker continues. Each message gets a child logger with the correlationId, the SQS message id and the try number.
5. Search. One Logs Insights query over the api and worker log groups, filtered by that id.

In MiniTickets

Log group	Written by	Kept
/minitickets/dev/api	api tasks (and the migration task)	7 days
/minitickets/dev/web	web tasks	7 days
/minitickets/dev/worker, /minitickets/dev/tickets	the two Lambdas	7 days

ECS tasks send their output with the awslogs driver; each task writes its own log stream. The Lambdas name their group in loggingConfig, so they do not use the default /aws/lambda/… group, which keeps logs forever. The api uses pino through Fastify; the worker writes the same JSON shape (level 30 info, 50 error; msg) straight to stdout, because console.log on Lambda would add a text prefix.

Three saved queries live in the folder minitickets-dev: follow-one-request, errors and worker-durations. No line ever holds an email address.

From → to	What happens	What allows it
Saved Logs Insights queries → CloudWatch log groupsonly when computeEnabled = true	the saved queries run over /minitickets/dev/api and /worker	config logGroupNames on each QueryDefinition; whoever runs a query needs logs:StartQuerycreateLogQueries

See the log groups on the map →

See it in AWS

Console

CloudWatch → Logs Insights → Saved queries → minitickets-dev/follow-one-request, paste the id from the browser’s network tab.

CLI

aws logs tail /minitickets/dev/api --since 10m

Code

apps/api/src/http/correlation-id.ts → correlationIdFor and infra/src/observability/log-queries.ts → createLogQueries.

Break it

Symptom	Cause	Where to look
follow-one-request finds api lines but no worker lines	The message lacked the id, or the worker logged with console.*	The worker’s raw lines
A new /aws/lambda/… group keeps growing	A Lambda without loggingConfig uses its default group	CloudWatch → Log groups
A task fails to start with an awslogs error	The log group in the task definition does not exist	ECS → stopped task

Say it in an interview

Say it like this (20-30 seconds)

Every api request gets a correlation id: the caller’s UUID if valid, otherwise a new one. Fastify writes it on every JSON log line and returns it in a header. The booking service copies it into the queue message, and the worker adds it to each line it writes about that message. Logs go to log groups with seven days of retention, and a saved Logs Insights query filters both groups by that one id.

Follow-ups

Why a random UUID and not Fastify's default ids?

The default restarts at req-1 in every process, so two api tasks would both have req-1. A UUID is unique across tasks and services.

How is this different from distributed tracing?

Tracing (for example AWS X-Ray or OpenTelemetry) also records timing per hop and draws the call tree. A correlation id is the cheap first step: it links lines, not timings.

What is the trade-off of 7 days of retention?

Storage stays cheap, but an issue found after a week has no logs. Production would keep longer, or export to cheaper storage.

Advanced: Details: the queries

errors shows every line with level >= 50, newest first. worker-durations reads Lambda’s own REPORT lines and shows calls, p95 and max duration, and memory in 5-minute buckets. The dashboard’s log box uses the same query text (infra/src/observability/dashboard-body.ts → ERRORS_QUERY).

Checkpoint

Answer these without scrolling up. If one feels hard, reread that section.

1. Where does the correlation id come from?
2. How does it reach the worker?
3. Why does the worker not use console.log on Lambda?
4. How long are log lines kept?
5. Which saved query would you run first after an alarm?

Redraw it from memory: browser → api (correlationId on every line, header back) → message.correlationId → SNS/SQS → worker (correlationId, sqsMessageId, tries) → one Logs Insights query

Try this: on the map, select the saved Logs Insights queries and see which log groups they read

Mark this page as read