koala73 / koala73/worldmonitor
test(e2e): investigate recurring map-budget smoke flakiness and Playwright response errors
- Dominant language
- TypeScript
- Stars
- 86.6k
- Forks
- 13.1k
- Avg merge
- 8h 4m
- Merged PRs (30d)
- 825
Description
## Problem
The required `variant-smoke (1)` CI job failed on PR #7835, then passed a fresh build of the same git tree with one flaky test. A later run passed without a reported retry. The cause remains unresolved.
Review `e2e/map-overlay-marker-budget.spec.ts`, specifically `keeps the full dashboard DOM and listener counts bounded across cold loads`. Distinguish a Playwright protocol failure from a real renderer-node regression or an unstable measurement. Do not assume these two signatures share a cause.
## Evidence from September 7, 2026
| Run and head | Observed result |
| --- | --- |
| [Failed job 101610952802](https://github.com/koala73/worldmonitor/actions/runs/34079140563/job/101610952802), `f336ad2f75264e9f11105cd5c73afd81472de970` | First attempt threw `Object with guid response@... was not bound in the connection`. The built-in retry reached the renderer-node assertion and measured **15,506**, above the **15,000** ceiling. Job ended with 58 passed and one failed. |
| [Fresh-build job 101611973151](https://github.com/koala73/worldmonitor/actions/runs/34079495998/job/101611973151), `1f24e746896ff17f506124b93a5239de463c8de1` | The response-object error recurred on the first attempt. The built-in retry passed. Job ended with **58 passed and one flaky**. |
| [Later job 101616325919](https://github.com/koala73/worldmonitor/actions/runs/34081068109/job/101616325919), `8f2edb13058cbd570a9594fcf3b0b9fe245dd577` | 59 passed, with no reported flaky test or retry. This head contains a weather metadata fix, not a browser-test repair. |
The first two heads have the identical git tree `3f08a12d2e52380f404117268b819757d6690798`. No renderer ceiling, browser runtime code, or retry count changed to obtain the passing fresh build.
The failing assertion is [rendererNodes at line 313](https://github.com/koala73/worldmonitor/blob/1f24e746896ff17f506124b93a5239de463c8de1/e2e/map-overlay-marker-budget.spec.ts#L313). The job ran `npm run test:e2e:ci-smoke:1`, with 59 tests and four workers. The failed run uploaded artifact `playwright-ci-smoke-1-34079140563-1`, including screenshots and a retry trace. Preserve relevant artifacts before retention expires.
## Investigation
1. Reproduce the cold-load test on both the affected head and its base, `60aa60b3e0660101eaae5cbbf324ff58220dcb8e`. Record exact Node, Playwright, Chromium, and runner versions.
2. Compare isolated execution with the full smoke shard at the CI worker count. Run a bounded repeated sample with retries disabled for diagnosis. Report failures and retries separately.
3. Inspect request routing, response-event handling, context creation and teardown, readiness, and renderer metric collection. Capture per-load DOM, renderer-node, and listener counts with timing.
4. Check whether detached nodes awaiting garbage collection, active dashboard work, or worker contention affect the measurement. Treat these as hypotheses, not established causes.
5. Identify each cause and implement the smallest verified repair. If the protocol error and renderer overrun are unrelated, record separate conclusions and link any required follow-up.
## Acceptance criteria
- [ ] Both failure signatures have an evidence-backed explanation, or an explicit remaining diagnostic gap. A passing rerun alone is not closure.
- [ ] Before-and-after evidence covers the exact cold-load case and the full smoke shard. Report sample size, first-attempt failures, retries, and metric ranges.
- [ ] The repair preserves detection of genuine DOM, renderer-node, and listener growth. Include a negative control that still fails when the relevant contract is broken.
- [ ] Repeated verification passes without relying on retries, followed by natural CI observations with a stated sample size.
- [ ] Do not close by raising the 15,000 ceiling, adding retries, skipping the test, swallowing protocol errors, or adding arbitrary waits. Any change to worker count or measurement semantics needs measured evidence and preserved regression coverage.
## Scope and related work
This issue tracks browser CI reliability. It does not authorize changes to seeders, production health budgets, or production configuration.
- #5685 is a closed report of the same response-object error signature. Its retry mitigation did not prevent the signature from recurring here.
- #6501 is an open browser-close investigation with a different observed signature. Cross-reference its evidence, but do not assume it is the same defect.
- #7112 introduced the dashboard DOM and listener budget coverage.
- #7835 exposed this incident. Its weather metadata repair does not establish a browser-flakiness fix.
Contributor guide
Research direction
Start with e2e/map-overlay-marker-budget.spec.ts, especially the cold-load test and its rendererNodes assertion at line 313. Reproduce it on the affected head and base with npm run test:e2e:ci-smoke:1, then compare isolated and full-shard runs while preserving the Playwright artifacts. Done means both failure signatures have evidence-backed conclusions, genuine budget regressions remain detectable, and repeated verification passes without retries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, playwright, typescript
- Domain
- ci-cd, testing-qa
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100