redhat-developer / redhat-developer/rhdh-plugin-export-overlays
Add AGENTS.md review guidance for test.skip removal PRs to cross-reference related issues
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 9
- Forks
- 72
- Avg merge
- 3d 11h
- Merged PRs (30d)
- 143
Description
What happened
PR #3338 removed a test.skip guard from the orchestrator E2E spec that was added by PR #3073 as a fix for issue #3068 (orchestrator backend crash). The review agent approved within 3 minutes, correctly noting the clean removal and the safe failure mode if the bug persists. However, it did not cross-reference the related open issue #3083 — the same crash in the bulk-import-orchestrator workspace — which also carries a test.skip and may now be ready for re-enablement.
What could go better
When a PR removes a test.skip that was added for a product_bug classification, the review agent could search for sibling issues filed for the same root cause across other workspaces. In this case, the skip message contained specific error details (TypeError in BackendInitializer.cjs.js:150) that could be used to find issue #3083. Surfacing stale related issues during review would help maintainers clean up test.skip guards and close resolved product_bug issues faster.
Confidence: Medium. The review agent's primary job is code quality assessment, and it performed that well. Cross-referencing related issues is an incremental improvement that adds a manual checkpoint. Existing proposal #3302 (auto-close product_bug issues after passing nightlies) would address the root cause more systematically — this proposal is complementary, not a substitute.
Proposed change
Add a subsection to the AGENTS.md 'E2E Nightly Fix Conventions' section titled 'Reviewing test.skip removal PRs'. The guidance should instruct reviewers that when a PR removes a test.skip added for a product_bug classification:
- Search open issues for the same root cause string from the skip message to identify related skips in other workspaces
- Note any related open issues in the review comment so maintainers can evaluate them for re-enablement
- Verify that the PR's test results confirm the underlying bug is resolved (e.g., E2E tests pass in the affected mode)
This is a lightweight addition (~10 lines) to the existing E2E conventions section, consistent with the pattern of other AGENTS.md review guidance issues (#2969, #3017, #3177).
Validation criteria
On the next PR that removes a test.skip for a product_bug classification, the review agent should: (1) identify and link related open issues with the same root cause, and (2) note whether those related issues may also be ready for re-enablement. Measure over the next 3 such PRs within 90 days.
Generated by retro agent from https://github.com/redhat-developer/rhdh-plugin-export-overlays/pull/3338
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in AGENTS.md at the “E2E Nightly Fix Conventions” section and review the related guidance issues #2969, #3017, and #3177 for the expected style. Add the requested review guidance for test.skip removals and ensure it covers searching sibling issues, mentioning them in review comments, and checking test results for re-enablement readiness.
Written by the indexing model from the issue text.
Assessment
- Domain
- documentation, testing-qa
- Issue type
- Documentation
- Difficulty
- 1/5
- Estimated time
- Under an hour
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 82/100