End-to-end load test

@marklise is already working on this.

Since Aug 28, 2026.

#### Description:
Simulate realistic peak (e.g. an opening-day release) through search -> hold -> book -> confirm, against a prod-like environment, with the waiting room release rate as the input.

This will gives us the number we actually need to configure the waiting room correctly - the release rate should be set from what the backend can sustain.

**Two load profiles, not one.** The Mode 2 gate covers only checkout-side routes (`/checkout`, `/reservation-flow`, `/cart`, `/facility`, `/booking-confirmation`, `/payment-retry`). `/search` and `/results` are **ungated**. So:

- **Ungated profile** — availability search takes the *full* 14,000-user burst, unthrottled by the waiting room. Search capacity must be sized against the whole crowd.
- **Gated profile** — hold → pay → confirm sees only the admitted release rate.

Running one blended profile at the release rate would under-test search by roughly the ratio of the burst to the release rate, so the two must run concurrently.

The path under test (front-door URLs, so the `/dayuse/api/*` strip function and its hop are in the measured path):

1. `POST /search` — availability search, OpenSearch-backed — **ungated**
2. `GET /product-dates/{collectionId}/{activityType}/{activityId}/{productId}` — availability detail
3. `POST /bookings` — creates the hold, decrements inventory, enforces waiting-room admission
4. `POST /transactions/{bookingId}` — payment handoff (Worldline; stubbed by default, see Notes)
5. Worldline notification webhook — inbound burst
6. `POST /bookings/{bookingId}/complete` — durable write, confirmation email, SMS enqueue
7. Async tail — DynamoDB Streams → OpenSearch reindex (`transDataStream` / `refDataStream`), SES, SNS

Scenarios to cover:

1. **Ramped arrival (gated path)** — step the rate (e.g. 10 → 25 → 50 → 100 completed bookings/min), holding each step to steady state. The last step within SLO is the number this ticket exists to produce.
2. **Full-burst search (ungated path)** — 14,000 users searching, run *concurrently* with scenario 1 so both contend for the same API and OpenSearch domain.
3. **Realistic peak** — an opening-day arrival pattern, fed at the release rate #680 can deliver, with the ungated search profile alongside.
4. **Contention** — a large share of load on a single facility/date/product, so inventory decrements land on one DynamoDB item.
5. **Abandonment** — 30–50% hold and never complete; held inventory releases on expiry and becomes bookable again under load.
6. **Cold start** — the burst against fully cold Lambdas, matching a scheduled opening.

#### Acceptance Criteria:

- **Given** a stepped arrival rate on the gated path, **when** the system reaches steady state at each step, **then** we can state the maximum sustained end-to-end bookings/min at which p95 latency stays within SLO and error rate stays under target. *(SLO and error target to be agreed with Product — proposal: p95 < 3s per step, < 0.5% 5xx.)*
- **Given** the ungated search profile at full 14,000-user burst, **when** run concurrently with the gated profile, **then** search stays within SLO, or the load at which it stops doing so is reported as a separate, explicit number.
- **Given** both numbers, **when** results are written up, **then** they are translated into concrete recommended `MAX_ACTIVE_SESSIONS` / `RELEASE_BATCH_SIZE` / `releaseIntervalSeconds` values for bcgov/reserve-rec-public#680.
- **Given** heavy contention on a single product/date, **when** many users book simultaneously, **then** inventory is never oversold, and conditional-write failure and retry rates are measured and reported.
- **Given** the burst, **when** DynamoDB Streams propagate to OpenSearch, **then** index lag is measured — users searching during the burst see stale availability and will attempt to book sold-out inventory. Report the lag and the resulting failed-booking rate.
- **Given** the peak, **when** confirmations fire, **then** SES send rate and SNS SMS throughput stay within account limits, or the required limit increases are identified.
- **Given** the peak completes, **then** no component degrades silently: Lambda throttles, DynamoDB throttles, OpenSearch queue rejections and API Gateway 429s are each reported at zero or explained.
- **Given** the measured worst-case checkout duration, **when** compared against `ADMISSION_TTL_MINUTES` and `product.holdDuration.minutes`, **then** confirm an admission token cannot expire mid-checkout.
- **Given** the run completes, **then** the bottleneck is named — which resource saturates first, and what raising it would take — and the result is explicitly recorded as a **shared** front-door/API ceiling, not per-product headroom.

#### Development Checklist:

- [ ] Choose and stand up the harness (recommend **k6** — ramping, thresholds, CI-friendly output). Nothing exists in either repo today
- [ ] Drive the front-door hostname with the `/dayuse` prefix so the strip function and extra hop are measured
- [ ] Build two concurrent profiles — ungated search at full burst, gated booking at the release rate — plus realistic browse ratio, abandonment and repeat searches
- [ ] Pre-mint a pool of Cognito test users / JWTs — booking requires authentication, and live token minting is itself rate-limited
- [ ] Seed a prod-like data set: enough collections / activities / products / inventory that search isn't hitting one hot document
- [ ] Stub the Worldline payment step by default; agree a separate small-scale burst against their sandbox to establish their ceiling
- [ ] Confirm WAF / rate-limit allowlisting for load-generator IPs
- [ ] Instrument: per-endpoint p50/p95/p99 and error rate; Lambda duration/errors/throttles/init duration; DynamoDB consumed capacity, throttles, conditional-check failures per table; OpenSearch CPU, JVM pressure, indexing lag; API Gateway 4xx/5xx/latency; CloudFront per-behaviour metrics; SES and SNS send rates
- [ ] Run scenarios 1–6
- [ ] Write up: sustained capacity number (gated), search ceiling (ungated), bottleneck, recommended waiting-room config, and any changes needed before release
- [ ] Establish a teardown/reset path so the run can be repeated after fixes

#### Dependencies

- Blocked by: prod-like test environment (matching Lambda memory and concurrency, OpenSearch instance sizing, DynamoDB capacity mode); Worldline decision (stub vs sandbox); WAF allowlist; seeded test data
- Blocking: waiting-room production configuration; release readiness sign-off
- Related to: bcgov/reserve-rec-public#680 — consumes this ticket's throughput number as its release-rate input
- Related to: #207 — unified front door; the capacity measured here is shared across every tenant mounted on it

**Relevant documentation as reference**

- `src/handlers/bookings/` (`POST/public.js`, `_bookingId/complete/POST/public.js`), `src/handlers/transactions/`, `src/handlers/search/`, `src/handlers/productDates/`, `src/handlers/worldlineNotification/`, `src/handlers/dynamoStream/`
- Front door routing and the `/dayuse/api/*` strip function: `lib/front-door-stack/front-door-stack.js` in reserve-rec-public

**Definition of Ready**

- [ ] Acceptance criteria are included
- [ ] Wireframes are included (if applicable)
- [ ] Design / Solution is accepted by Product Owner (if applicable)
- [ ] Dependencies are identified (technical, business, regulatory/policy)
- [ ] Story has been estimated (under 13 pts)

**Definition of Done**
- In progress:
- [ ] Acceptance criteria are tested (Functionality meets the acceptance criteria defined in the ticket)
- [ ] UI meets accessibility requirements
- [ ] Unit tests are written
- [ ] Work is traceable in GitHub
- [ ] PR linked to ticket number
- [ ] If needed/required - Dev adds flag/label to highlight any migration steps necessary prior to PROD deployment
- Code review:
- [ ] Code is peer reviewed and has passed CI/CD tests
- QA:
- [ ] Acceptance criteria are tested (Functionality meets the acceptance criteria defined in the ticket)
- [ ] Code is potentially shippable to the production environment
- [ ] Functional features have been tested and passed by QA
- [ ] UI components tested by designer
- [ ] Code is deployed to PROD when moved to 'done' column (unless requested otherwise by PO)
- PO Review:
- [ ] Acceptance criteria are tested (Functionality meets the acceptance criteria defined in the ticket)
- [ ] Reviewed and approved by Product Owner

#### Notes:

- **Environment fidelity is the whole ballgame.** A number measured against a scaled-down dev environment is not usable for configuring production. If a prod-like environment isn't available, that needs saying before this is estimated.
- **This is a shared ceiling.** The front door is a multi-product entry point — day-use is tenant #1 at `/dayuse`, and a future tenant (e.g. `/backcountry`) would strip to the same `/api/*` on the same API Gateway origin. The number produced here is total capacity across all tenants, not day-use headroom. It should be re-measured when a second product mounts.
- **Worldline is a third party** — we can't load-test their production endpoint without agreement. Default plan: stub the payment call to measure our own throughput, then a small separate burst against their sandbox to confirm their limits and the notification webhook's inbound capacity.
- **Fail-open risk:** the waiting-room check in `POST /bookings` catches lookup errors and falls through to `waitingRoomActive = false`. Under load, a throttle on the waiting-room table would silently disable queue enforcement. Worth measuring rather than assuming — and note there is no Lambda@Edge backstop on the front door's `/dayuse/api/*` behaviour, so origin enforcement is the only layer.
- **Cold-start amplification:** no reserved or provisioned concurrency is configured. At a scheduled opening every container is cold, and each fetches the HMAC key from Secrets Manager on first invocation (cached per container, not globally). Scenario 6 exists to measure this.
- **Single-item contention** is the likeliest hard failure: an opening-day release concentrates thousands of users on one facility/date, so inventory decrements and the queue's `admittedCount` counter all land on individual DynamoDB items.