redhat-developer / redhat-developer/rhdh-plugin-export-overlays

Add AGENTS.md guidance for test.runOnce key scoping in multi-project workspaces

Open Beginner friendly
#3,333 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

documentation ready-for-triage ready-to-code stale
Dominant language
TypeScript
Stars
9
Forks
72
Avg merge
3d 11h
Merged PRs (30d)
143

Description

What happened

PR #3318 added a bulk-import-app-next Playwright project that runs the same spec file as the existing bulk-import project. The spec used test.runOnce("bulk-import-rhdh-setup", ...) with a hardcoded key. Because runOnce flags are process-scoped and keyed by string alone in a shared /tmp/ directory, whichever project ran second would silently skip configure() + deploy(), leaving no RHDH deployment in that namespace. All 9 tests would fail with a misleading "missing heading" error.

The review agent caught this on the first review (commit edae03f, $1.21, 5m50s) and requested changes. The author fixed it in 3 minutes by scoping the key with ${rhdh.deploymentConfig.namespace}, matching the pattern deploy() uses internally.

The PR author explicitly noted: "Worth noting for the remaining migrations: none of the five already-migrated workspaces use runOnce, which is why no earlier NFS migration surfaced this. Any workspace that does — and runs one spec in both shells — has the same trap."

What could go better

The review agent caught this bug, which is the right backstop — but a code agent performing a future NFS migration would likely reproduce the same mistake if it copies from the pre-fix version of this spec or another workspace that uses runOnce with a literal key. AGENTS.md already documents test.runOnce usage patterns (including the deploy() internal guard and when runOnce is vs. isn't needed), but it does not warn about the key collision trap when the same spec runs in multiple projects.

Confidence: high. The AGENTS.md E2E section is actively consumed by both code and review agents (evidenced by the review agent's cross-workspace comparison in this very PR), and similar guidance issues have been filed and implemented before (#3112, #2997, #3216).

Proposed change

Add a subsection under the existing test.runOnce documentation in AGENTS.md (after the "Key rules" bullet list) warning about key collisions in multi-project workspaces:

Content to add:

  • When the same spec file runs in multiple Playwright projects (e.g. bulk-import and bulk-import-app-next), a literal runOnce key creates a collision — the first project's flag satisfies the second, silently skipping its deployment.
  • Always scope runOnce keys by namespace: `workspace-setup-${rhdh.deploymentConfig.namespace}`. This mirrors what deploy() does internally (deploy-${namespace}).
  • The failure mode is silent: all tests in the second project fail on a missing UI element with no error pointing to the skipped deployment.
  • Reference PR #3318 as the case study.

File: AGENTS.md, in the E2E Testing > test.runOnce section.

Validation criteria

The next code agent PR that adds an -app-next project to a workspace using test.runOnce should scope the key by namespace from the start, without requiring a review agent catch-and-fix cycle. Measurable over the next 3 NFS migration PRs that involve runOnce.


Generated by retro agent from https://github.com/redhat-developer/rhdh-plugin-export-overlays/pull/3318

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Open AGENTS.md and locate the E2E Testing > test.runOnce section, then review PR #3318 for the multi-project collision case. Done means the section explains namespace-scoped keys, the silent skipped-deployment failure mode, and references PR #3318 so future migrations avoid the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
playwright, typescript
Domain
documentation, testing-qa
Issue type
Documentation
Difficulty
1/5
Estimated time
Under an hour
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
92/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.