galaxyproject / galaxyproject/foundry
Prove the IWC review pipeline on a real PR and accept its policy pin
- Dominant language
- TypeScript
- Stars
- 2
- Forks
- 3
- Avg merge
- 1d 22m
- Merged PRs (30d)
- 63
Description
> **Posted by Claude (AI assistant) on behalf of @jmchilton.** Not authored by them personally.
Follow-up to #491, which is closed as structurally complete. #509 landed the `review-galaxy-workflow` Mold, the four-phase `GALAXY WORKFLOW REVIEW` Pipeline, the `galaxy-workflow-validation-result` output on `validate-galaxy-workflow`, the honest `test-definition-missing` / `not-run` handoff on `run-workflow-test`, both eval tiers, and seven fixture directories. What did not land is the evidence, and the policy the review applies is still a proposal rather than accepted upstream.
This is the review-side twin of #531.
## The gap
`content/pipelines/galaxy-workflow-review/scenarios.md` states it under "Tier maturity — what would gate the walk":
> **Nothing has been run.** No phase has executed end to end, no eval property has ever been scored, and every fixture under `content/molds/review-galaxy-workflow/examples/` is hand-authored rather than harvested from a real run. The issue's own task — exercise this on at least one real IWC or IWC-Lab pull request — is the next move, and the two harness-gate cases above can only be walked there.
> **The policy pin is proposed, not accepted.** galaxyproject/iwc#1366 is still open, so the vendored review command is a proposed revision.
That second point is the sharper one. `content/prompts/iwc/workflow-pr-review-command/upstream.prompt` is pinned to `jmchilton/iwc@d7012eb` — 134 lines, the expansion proposed in galaxyproject/iwc#1366. The `.claude/commands/review.md` actually on `galaxyproject/iwc` today is 60 lines and differs. So the Mold reports provenance for a revision no one upstream has accepted, and #491's acceptance criterion "the review uses the accepted, pinned upstream IWC prompt and reports its provenance" is not yet satisfiable.
## Scope
1. **Land the upstream policy chain.** galaxyproject/iwc#1366 merges, then re-pin `workflow-pr-review-command` to the accepted `galaxyproject/iwc` commit and refresh `copilot-workflow-review-instructions` off `b80bc92780089b54773f558c224a27b38baa8a06`. #509 put the refresh procedure in one place — `content/prompts/iwc/workflow-pr-review-command/index.md` §Provenance and acceptance status — so this is re-vendor, update three SHA sites, update that section, `make casts`.
2. **Record the semantic-superset finding.** Compare the accepted Claude command against the accepted Copilot instructions and write the result down. `copilot-workflow-review-instructions/index.md` currently claims only that the two "substantially overlap"; #491 asks for a superset determination, and for the difference to be surfaced rather than dropped if it is not one.
3. **Exercise the Pipeline on a real IWC or IWC-Lab PR.** All four phases, end to end, with the review retained. This is #491's last implementation task and the only place the eval properties get scored for the first time.
4. **Walk the two harness gates there.** SHA mismatch stops before phase 3; an unconfirmed checkout stops before phase 3 rather than defaulting to trusted. Both are stated as oracles in `scenarios.md` because `galaxy-workflow-pr-context` has no producing Mold — a real run is the only thing that checks them.
5. **Run `check:vendored` once.** It compares against local upstream checkouts named in `common_paths.yml`, CI does not run it, and it has never been run against these prompts. The re-pin in (1) is exactly when it matters.
6. **Even out the thin fixtures.** `label-test-mismatch/`, `missing-test/`, and `planemo-failure/` lack the `.dockstore.yml`, `README.md`, `CHANGELOG.md`, and pull-request context that `clean-passing/` carries, so each differs from the baseline in more than its one intended variable. Their expectations survive that today, but the isolation is weaker than the `with-foundry-context` / `clean-passing` pair's.
## Out of scope
Everything #491 deferred — splitting into separate collection/IWC/Foundry/consolidation Molds, JSON evidence or report schemas, stable finding fingerprints, a deterministic Markdown renderer, a GitHub comment or check reporter, hosted execution of untrusted fork heads, multi-workflow fan-out. Also out of scope: changing the review checklist. If a real run shows the Mold gets something wrong, that is a separate issue against the Mold.
## Blocked on
galaxyproject/iwc#1366, which is open and last saw activity on 2026-09-09. Items 3 through 6 can proceed against the current pin, but the run in (3) is worth more once the policy is the accepted one — a review scored against a proposal has to be scored again.
## Done when
- The vendored review command is pinned to an accepted `galaxyproject/iwc` commit and the Mold reports that provenance.
- The Claude-versus-Copilot superset finding is recorded in the Prompt framing note.
- The Pipeline has run end to end on at least one real IWC or IWC-Lab pull request, with the review retained.
- Both harness gates have been observed stopping the run before phase 3.
- `check:vendored` has been run against the re-pinned prompts.
- `review-galaxy-workflow` and `galaxy-workflow-review` can move off `status: draft` on the strength of that evidence.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with content/pipelines/galaxy-workflow-review/scenarios.md and content/prompts/iwc/workflow-pr-review-command/index.md §Provenance and acceptance status, then check galaxyproject/iwc#1366 and run check:vendored against the local upstream checkouts. The work is done when the accepted prompt is re-vendored, the superset finding is recorded, a real PR exercises all phases and harness gates, and the evidence supports moving both artifacts out of draft.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- git, github
- Domain
- devtools, documentation, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100