Slow first requests: cold starts

Symptom

Troubleshooting scenario: the first worker invocation after idle takes longer; its Lambda REPORT includes “Init Duration:”. Later invocations are faster. A cold start initializes a new execution environment; it can also happen during scale-out, not just after idle.

Read the error first

Lambda reports initialization, not an error or IAM refusal. Under Reach / Allowed / Password, timing alone proves none of them failed. A dependency can also be slow on a warm run; an AccessDenied still needs missing-allow/explicit-deny analysis. MiniTickets’ API/web run on ECS, so a slow /events page is not automatically a Lambda cold start.

Decide

flowchart TB
  A["Slow first invocation"] --> B{"Initialization visible?"}
  B -->|"yes"| C["Compare init with total delay"]
  B -->|"no"| D["Inspect dependency calls and queue age"]
  C --> E{"Warm runs fast?"}
  E -->|"yes"| F["Measure startup changes against latency needs"]
  E -->|"no"| D

Likely causes

Ranked for first-only slowness with an initialization field.

1. Runtime and application initialization in a new environment.
2. Lazy first-use work, such as the tickets worker fetching its signing secret.
3. Queue wait or downstream latency coincides with startup. These also affect warm runs and require separate timing evidence.

How to check

Console

In /minitickets/dev/worker or /minitickets/dev/tickets, compare REPORT initialization and duration fields for several invocation ids. Compare Lambda Duration and SQS oldest-message age. Separate enqueue-to-start delay from work inside the handler.

CLI

Read deployment and compute settings without exposing environment variables:

export AWS_PROFILE=ticketlite
aws lambda get-function-configuration --function-name minitickets-dev-worker --query '{Memory:MemorySize,Timeout:Timeout,Architecture:Architectures,Modified:LastModified}'
aws lambda list-provisioned-concurrency-configs --function-name minitickets-dev-worker

CloudTrail

UpdateFunctionCode and UpdateFunctionConfiguration show deployment changes around the slowdown. They do not record every cold start or its duration; use Lambda logs and metrics. An invocation data event alone also does not explain its initialization cost.

Fix

Preserve SDK clients outside the handler, as current code does. Measure startup/dependency changes and memory settings with representative batches. These asynchronous workers tolerate queue delay; provisioned concurrency, paid preinitialized capacity, is a future option only if measured latency needs justify it. Reserved concurrency does not preinitialize environments.

Prevent

Record separate cold/warm measurements after worker changes; watch end-to-end ticket/email delay. Do not promise a warm environment survives idle or that every first request has identical latency.

apps/worker/src/lambda.ts → handler; Inspect the tickets worker →.

Say it in 30 seconds

Say it like this (20-30 seconds)

I prove a cold start with initialization evidence and compare it with warm handler durations and queue age. Our Lambda workers are asynchronous, while the web and API run on ECS, so I first identify which path is slow. SDK clients already live outside the handler for reuse. I measure initialization, lazy secret loading and memory changes before considering paid provisioned concurrency. Reserved concurrency limits capacity; it does not keep environments warm.

Advanced: Historical measurement and logging limits

The local runtime-emulator run recorded in docs/40-aws-job-pipeline.md on 2026-10-15 had INIT 302 ms and a warm batch of 23 ms. These are local historical measurements, not current AWS latency promises, and the two numbers measure different phases. Neither function currently configures provisioned concurrency. Both use the worker image; email has 256 MB, tickets 512 MB. The signing-secret helper caches only successful reads in a warm process. AWS’s execution lifecycle also explains initialization after failures: a reset can cause a suppressed initialization inside the next reported duration. Absence of an obvious init field is not universal proof of a warm run. apps/worker/src/tickets/assets.ts → signingSecret; infra/src/tickets/lambda.ts → createTicketLambda.