redhat-developer / redhat-developer/rhdh-plugins

Add release workflow and issue-scope context to agent review instructions

Open Beginner friendly
#4,507 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

documentation fullsend ready-for-triage ready-to-code Review effort 1/5
Dominant language
TypeScript
Stars
48
Forks
120
Avg merge
2d 6h
Merged PRs (30d)
337

Description

What happened

On PR #4417, the review agent produced two categories of false positives across 4 review rounds (~$28 total). First, in review round 2, the agent surfaced a [medium] scope-completeness finding claiming the PR could not close issue #4046 because npm publish and CHANGELOG tasks were incomplete. The human explained that npm publish is handled by the repo's automated Version Packages workflow, not individual PRs, and the changeset entry already existed. Second, across all 4 rounds, the agent repeatedly flagged stale references in documentation files (design.md, proposal.md, annotation-scheme/spec.md) that were outside the issue's authorized scope. The human documented a clear scoping rule: only files cited by issue #4046 should be updated in this PR.

What could go better

The review agent lacks two pieces of repo-specific context that caused repeated noise. (1) It does not know the repo uses changeset-based automated releases via Version Packages PRs, so it incorrectly flags missing npm publish steps as incomplete scope. (2) It does not understand that PRs may intentionally limit changes to files and components cited in the linked issue, so it flags stale references in uncited files as defects rather than informational observations. Confidence: high for the release workflow gap (clearly documented in the human's triage). Moderate for the scope convention — this may be a team preference rather than a hard rule, and different PRs may have different scope policies. These findings re-appeared across rounds, which is separately tracked by fullsend-ai/fullsend#3073 and #3515.

Proposed change

Add two context paragraphs to .fullsend/AGENTS.md. (1) Under a 'Release workflow' heading: "This repo uses changesets for versioning. Individual PRs include a .changeset/*.md entry declaring the bump level. npm publish is handled automatically by the Version Packages workflow — do not flag PRs for missing publish or CHANGELOG steps." (2) Under a 'Review scope' heading: "When a PR references a specific issue (Closes #XXXX), changes may be intentionally limited to files and components cited in that issue. Stale references or inconsistencies in uncited files should be noted as informational observations, not flagged as defects requiring fixes in the current PR."

Validation criteria

The next 5 review agent runs on PRs in this repo that reference specific issues should produce zero false positives about missing npm publish steps or Version Packages workflow, and should distinguish in-scope defects from out-of-scope informational observations in their findings.


Generated by retro agent from https://github.com/redhat-developer/rhdh-plugins/pull/4417

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Edit .fullsend/AGENTS.md, adding the proposed Release workflow and Review scope context paragraphs. Check the existing instruction structure first, then validate the change against the stated criteria: the next five review-agent runs should avoid false positives about npm publishing and distinguish out-of-scope observations from defects.

Written by the indexing model from the issue text.

Assessment

Domain
documentation
Issue type
Documentation
Difficulty
1/5
Estimated time
Under an hour
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
85/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.