Bound release Playwright smoke and report the five-minute verification budget

Product outcome

For an ordinary successful release, the always-on local Playwright smoke tier is a deliberately small boot/auth/management canary rather than a broad product matrix. The scheduled full suite retains every moved scenario. The existing scheduled selection-observability artifact also reports whether eligible ordinary main pushes complete their release verification critical path within 5m00s.

This is #228 Slice B. Slice A remains shipped and accepted at feature commit 109a108b33f106029530dda95fbd85896dd084c7, merged as 373e77e920e3978c718d6bde2921a2d01581548b; this issue must not reopen or weaken it.

Normative authority

• _docs/PROCESS.md: issue lifecycle, proportional Playwright tiers, evidence, and on-call ownership.
• _docs/ci/change-selective-ci.md: exact range selection, smoke/core/full policy, force-full rules, four-hour full backstop, bounded evidence, and release gate.
• _docs/ci/selection-observability.md: bounded latest-20 main-push aggregate and redaction.
• _docs/ci/playwright-flake-policy.md: no automatic retries; complete terminal counts and quarantine separation.
• _docs/specs/10-verification-strategy.md: release-critical local browser coverage and separate scheduled/deployed layers.
• Parent #228, especially the accepted Slice A evidence and PM child disposition.

Current evidence

• make test-playwright-smoke currently selects 30 cases, mainly because whole files are marked smoke: test_foundation_smoke.py, test_single_identity.py, and test_studio_foundation.py.
• Push run 32963147116 for Slice A recorded a 2m44s Playwright job, but about 9m40s elapsed from classification start through ci-gate. The deploy later failed, so this is not a green five-minute sample.
• ci.history currently aggregates profile/reason/evidence-reuse facts for at most 20 completed ci.yml main push runs. It does not report job timing.
• Remote main remains face8e4808d65afbf0374d1ced7a88079950d663. Its latest scheduled full regression, run 33315212334, is terminal red across the open recovery owners: projection digests (#253), pagination/migration/Django (#261), offline Spotify embeds (#279), podcast URL/resource behavior (#223/#270), and the graph target (#236). #232 test-only reconstruction and #236 acceptance remain open. This is release-blocker evidence, not an eligible #238 SLO sample.

Foundation and ownership reconciliation

• Closed #210 is the accepted smoke/core/full marker, Make-target, safety-exclusion, and planner-profile vocabulary. Its historical counts are not current inventory evidence and #238 does not reopen it.
• Accepted #228 Slice A is the known-owner/range foundation: backend-only known-owner impact selects smoke, ordinary render impact selects core, and template/browser-harness or unknown unsafe impact selects full. #238 changes the reviewed smoke membership and timing observability only; it must not alter those mappings, classifier range semantics, or ownership closure.
• #252 is an independent P0 full-suite capacity sibling, not a prerequisite or substitute. After the common recovery release, normal lowest-numbered P0 selection gives #238 precedence unless the product owner directs otherwise. The lanes must not edit overlapping Make/workflow/Playwright paths concurrently; the later lane rebases and regenerates its plan.

Scope

1. Bound the smoke inventory

1. Define an explicit smoke inventory of 1–12 collected Playwright node IDs, including parameterizations.
2. Keep only release-canary behavior:
   • one anonymous public application/bootstrap path;
   • one safe signed-out authentication/return-intent path;
   • one capability-protected Studio entry/denial path;
   • the minimum desktop/mobile parameterization needed to prove those canaries.
3. Reclassify every scenario removed from smoke as core or full so it remains selected by make test-playwright and the scheduled full regression. No test is deleted or silently unmarked.
4. Preserve the existing safety exclusions, flake-policy plugin, no-retry rule, complete-count evidence, 600-second outer bound, and fail-closed collection behavior.
5. Add marker-registry/Make/workflow tests that pin the exact smoke node inventory and prove copied e2e/, remote_readonly, remote_mutation, live_email, live_provider, and quarantined tests are absent.
6. Treat the required playwright_tests/ marker edits as the test-harness/render impact that the existing ownership graph declares. Do not add an ownership, planner, or screenshot exception for this issue. The versioned plan must select fresh full Playwright and graph-derived independent screenshots even though application render bytes are not expected to change.

2. Add release-budget observability

Extend the existing scheduled latest-20 ci.yml main-push report. For each validated completed push, consume bounded GitHub run/job timing plus the exact selection/plan artifact.

An eligible SLO sample must satisfy all of:

• event push, branch main, completed run and aggregate conclusion success;
• validated selection is documentation/documentation_only or focused/single_application;
• no force-full risk or full profile;
• jobs named classification and ci-gate each completed successfully with well-formed timestamps and non-negative ordering.

Define:

• verification_seconds = ci-gate.completed_at - classification.started_at;
• within budget iff verification_seconds <= 300;
• delivery_seconds = deploy.completed_at - run.created_at only when that same eligible run has a successful deploy; delivery is a separate descriptive series and never substitutes for the five-minute verification result.

The aggregate must expose, without per-run rows:

• runs considered and eligible verification sample count;
• latest, median, and maximum verification seconds;
• count and rate at or below 300 seconds;
• successful-delivery sample count plus latest, median, and maximum delivery seconds;
• empty when the bounded history truly has no runs;
• unavailable when required history/artifacts/timestamps are missing, malformed, oversized, ambiguous, or inaccessible.

Use a deterministic median (middle value after numeric sort; arithmetic mean of the two middle values for an even count). Durations are finite and non-negative. An eligible sample must come from an exact workflow/controller revision carrying the bounded critical-path timeout contract below; older runs without that contract are not eligible evidence.

3. Bound the measured workflow interval

The measured classification-start through gate-completion interval must be bounded by explicit job timeouts. Preserve the existing classification: 10, quality: 15, django: 30, playwright: 60, and screenshots: 30 minute bounds, and add only:

• container.timeout-minutes: 60;
• ci-gate.timeout-minutes: 10.

The resulting validation ceiling is 4,800 seconds: 10 minutes for classification, plus the longest 60-minute parallel verification job, plus 10 minutes for the aggregate gate. A larger, reversed, non-finite, or otherwise impossible interval is unavailable, not an SLO miss or pass. The success threshold remains exactly 300 seconds. The timeout additions may terminate a wedged job and therefore fail the release normally; they must not change job dependencies, conditions, permissions, commands, evidence requirements, image semantics, or gate verdict logic.

Dependencies and readiness

Depends on #232 and #236. Do not start engineering while either is open.

Before those issue-local dependencies can close, the shared recovery baseline must complete in order #261 → #253 → #279 → #223 → #270, including the external-source prerequisite and re-grooming required by #253. A changed base invalidates prior plans and evidence.

Readiness requires this exact sequence:

1. #232 receives tester PASS and PM acceptance on the ordered combined tree.
2. #236 receives tester PASS and PM acceptance on that exact ordered tree.
3. Their focused commits merge in #232 then #236 order and are pushed as one accepted main state.
4. That exact combined release SHA records playwright, screenshots, ci-gate, publish, and deploy successful, and the deployed host reports the sealed identity healthy/ready.

The first natural eligible push after this issue lands supplies the empirical SLO gate. Do not dispatch, rerun, or create a no-op commit solely to manufacture a sample.

Non-goals

• No change to classifier range semantics, ownership closure, evidence reuse, quality, Django selection, screenshot policy, container/image semantics, publish/deploy authorization, or ci-gate logic. The only allowed workflow-bound changes are the explicit container: 60 and ci-gate: 10 minute job timeouts above.
• No weakening of render-driven core/full Playwright, template/browser-harness full fallback, unknown/unsafe full fallback, scheduled make test-playwright, quarantine monitoring, or remote/live safety markers.
• No application, template, CSS, JavaScript, URL, auth, Studio, API, database, or product behavior change.
• No test deletion, xfail, retry, quarantine, assertion weakening, or conversion of failed/missing history into a pass.
• No claim that deploy duration is within the five-minute verification SLO.
• No workflow run or deployment initiated by engineering/testing for measurement.

Acceptance criteria

Smoke contract

• make test-playwright-smoke collects between 1 and 12 exact cases, including parameterizations, and every collected node belongs to the reviewed public/auth/Studio canary inventory.
• Every case removed from smoke is explicitly core/full and remains present in full local and scheduled collection; total blocking full-suite coverage is not reduced.
• Smoke excludes quarantine and every remote/mutation/email/provider marker and cannot collect copied e2e/ tests.
• Smoke produces one complete flake-policy record with attempted = passed + failed + skipped, rerun=0, and a matching terminal pytest summary; partial/timeout/malformed output remains failure.
• Marker registry, Make target, selection, workflow, and evidence tests pin the inventory and prevent accidental growth or unsafe collection.

Observability contract

• The bounded latest-20 report validates exact main-push selection artifacts and GitHub job timestamps before counting a sample.
• Eligibility, the classification-start → gate-complete interval, 300-second threshold, and separate creation → deploy-complete interval match the definitions above.
• Aggregate schema validation covers odd/even medians, boundary 300, zero eligible samples, mixed eligible/ineligible runs, failed/cancelled/skipped jobs, missing/duplicate jobs, invalid ordering, oversized/malformed artifacts, and API/history failure.
• The report and Actions summary expose only aggregate counts/rates/durations. They contain no run ID, attempt, SHA, path, artifact URL/name, job URL, log, credential, environment secret, or production datum.
• Missing or untrusted required evidence yields unavailable; it never authorizes narrowing or reports the SLO as met.
• Existing profile/reason/reuse observability remains correct and scheduled selection still fails closed independently of observability availability.
• Workflow tests pin container.timeout-minutes: 60 and ci-gate.timeout-minutes: 10; runs whose exact controller lacks either bound are excluded from eligible timing samples, and intervals above 4,800 seconds are unavailable.

Verification and lifecycle

• Focused history/marker/workflow/evidence tests, make test-ci, lint, format, typecheck, security, and repository workflow contracts pass.
• The versioned verification plan classifies all components exactly once and, because playwright_tests/ marker inputs change, records render/test-harness impact, fresh full Playwright, and required graph-derived screenshots. No planner or ownership exception is introduced to force not_applicable.
• An independent tester verifies exact smoke collection, retained full collection, timing schema/redaction/fail-closed cases, all selected repository gates, and every plan-required desktop/mobile screenshot. The images must be captured from the unchanged product surfaces, opened, and confirmed readable, contained, and free of error/debug pages.
• PM accepts the frozen automated candidate from the release-operator perspective; this authorizes the focused Refs #238 commit but does not claim the later natural SLO sample.
• [ON-CALL] The first naturally occurring eligible successful main push records a valid sample. Its verification interval is at most 300 seconds; delivery duration is reported separately. A red or unavailable sample returns through the lifecycle and is not called green.

Scenarios

1. Repository — smoke inventory: collect smoke, core, and full. Smoke is 1–12 reviewed nodes; every moved node remains in core/full; unsafe markers and copied e2e are absent.
2. Repository — exact threshold: synthetic eligible records at 299, 300, and 301 seconds report two within budget and one outside.
3. Repository — eligibility: documentation and one-app successful runs count; full/unknown/shared/migration, failed, cancelled, or incomplete runs do not.
4. Repository — fail closed: missing gate job, duplicate classification job, reversed timestamp, malformed archive, excessive response, or API failure makes timing observability unavailable without changing scheduled regression selection.
5. Operations — green ordinary push: classification and affected jobs run in parallel; the gate completes within five minutes; later publish/deploy duration is separate.
6. On-call — miss: a valid eligible push above five minutes but within the 4,800-second workflow ceiling is reported as an SLO miss with aggregate-safe evidence and routed back to #238; no test or gate is bypassed.
7. Tester — plan-derived render evidence: the marker-only candidate selects full Playwright and its graph-derived desktop/mobile screenshot matrix; the tester opens every capture and proves the unchanged application surfaces remain readable and error-free.

Lifecycle handoff

After dependencies close, engineer implements in an isolated worktree without commit/push/workflow dispatch, generates the versioned plan, runs the selected local gates, and freezes. Independent tester recomputes the plan and verifies the exact collections, synthetic timing/redaction cases, full Playwright, and required screenshots. PM accepts the automated candidate. The focused commit uses Refs #238; the issue remains open, and human is added only if the sole remaining gate is the natural hosted SLO observation. Orchestrator merges/pushes, and on-call alone observes the first natural eligible sample. A valid sample at or below 300 seconds permits final closure; a miss, unavailable result, red release, or absence of an eligible natural push keeps the issue open and returns actionable failure through the normal lifecycle.

Begin only after #232 and #236 are accepted on the required combined tree. Read _docs/ci/change-selective-ci.md and _docs/ci/selection-observability.md, then inspect the ci.yml workflow, the existing ci.history aggregation, playwright_tests/ marker registry, and make test-playwright-smoke. Done means the reviewed 1–12-case smoke inventory, bounded timing aggregate, workflow timeout contracts, and all listed focused and repository checks pass.