galaxyproject / galaxyproject/foundry

Prove the IWC review pipeline on a real PR and accept its policy pin

Open
#535 0 comments 0 reactions 0 assignees View on GitHub
agent/implement enhancement priority/mvp roadmap/main
Dominant language
TypeScript
Stars
2
Forks
3
Avg merge
1d 22m
Merged PRs (30d)
63

Description

> **Posted by Claude (AI assistant) on behalf of @jmchilton.** Not authored by them personally.

Follow-up to #491, which is closed as structurally complete. #509 landed the `review-galaxy-workflow` Mold, the four-phase `GALAXY WORKFLOW REVIEW` Pipeline, the `galaxy-workflow-validation-result` output on `validate-galaxy-workflow`, the honest `test-definition-missing` / `not-run` handoff on `run-workflow-test`, both eval tiers, and seven fixture directories. What did not land is the evidence, and the policy the review applies is still a proposal rather than accepted upstream.

This is the review-side twin of #531.

## The gap

`content/pipelines/galaxy-workflow-review/scenarios.md` states it under "Tier maturity — what would gate the walk":

> **Nothing has been run.** No phase has executed end to end, no eval property has ever been scored, and every fixture under `content/molds/review-galaxy-workflow/examples/` is hand-authored rather than harvested from a real run. The issue's own task — exercise this on at least one real IWC or IWC-Lab pull request — is the next move, and the two harness-gate cases above can only be walked there.

> **The policy pin is proposed, not accepted.** galaxyproject/iwc#1366 is still open, so the vendored review command is a proposed revision.

That second point is the sharper one. `content/prompts/iwc/workflow-pr-review-command/upstream.prompt` is pinned to `jmchilton/iwc@d7012eb` — 134 lines, the expansion proposed in galaxyproject/iwc#1366. The `.claude/commands/review.md` actually on `galaxyproject/iwc` today is 60 lines and differs. So the Mold reports provenance for a revision no one upstream has accepted, and #491's acceptance criterion "the review uses the accepted, pinned upstream IWC prompt and reports its provenance" is not yet satisfiable.

## Scope

1. **Land the upstream policy chain.** galaxyproject/iwc#1366 merges, then re-pin `workflow-pr-review-command` to the accepted `galaxyproject/iwc` commit and refresh `copilot-workflow-review-instructions` off `b80bc92780089b54773f558c224a27b38baa8a06`. #509 put the refresh procedure in one place — `content/prompts/iwc/workflow-pr-review-command/index.md` §Provenance and acceptance status — so this is re-vendor, update three SHA sites, update that section, `make casts`.
2. **Record the semantic-superset finding.** Compare the accepted Claude command against the accepted Copilot instructions and write the result down. `copilot-workflow-review-instructions/index.md` currently claims only that the two "substantially overlap"; #491 asks for a superset determination, and for the difference to be surfaced rather than dropped if it is not one.
3. **Exercise the Pipeline on a real IWC or IWC-Lab PR.** All four phases, end to end, with the review retained. This is #491's last implementation task and the only place the eval properties get scored for the first time.
4. **Walk the two harness gates there.** SHA mismatch stops before phase 3; an unconfirmed checkout stops before phase 3 rather than defaulting to trusted. Both are stated as oracles in `scenarios.md` because `galaxy-workflow-pr-context` has no producing Mold — a real run is the only thing that checks them.
5. **Run `check:vendored` once.** It compares against local upstream checkouts named in `common_paths.yml`, CI does not run it, and it has never been run against these prompts. The re-pin in (1) is exactly when it matters.
6. **Even out the thin fixtures.** `label-test-mismatch/`, `missing-test/`, and `planemo-failure/` lack the `.dockstore.yml`, `README.md`, `CHANGELOG.md`, and pull-request context that `clean-passing/` carries, so each differs from the baseline in more than its one intended variable. Their expectations survive that today, but the isolation is weaker than the `with-foundry-context` / `clean-passing` pair's.

## Out of scope

Everything #491 deferred — splitting into separate collection/IWC/Foundry/consolidation Molds, JSON evidence or report schemas, stable finding fingerprints, a deterministic Markdown renderer, a GitHub comment or check reporter, hosted execution of untrusted fork heads, multi-workflow fan-out. Also out of scope: changing the review checklist. If a real run shows the Mold gets something wrong, that is a separate issue against the Mold.

## Blocked on

galaxyproject/iwc#1366, which is open and last saw activity on 2026-09-09. Items 3 through 6 can proceed against the current pin, but the run in (3) is worth more once the policy is the accepted one — a review scored against a proposal has to be scored again.

## Done when

- The vendored review command is pinned to an accepted `galaxyproject/iwc` commit and the Mold reports that provenance.
- The Claude-versus-Copilot superset finding is recorded in the Prompt framing note.
- The Pipeline has run end to end on at least one real IWC or IWC-Lab pull request, with the review retained.
- Both harness gates have been observed stopping the run before phase 3.
- `check:vendored` has been run against the re-pinned prompts.
- `review-galaxy-workflow` and `galaxy-workflow-review` can move off `status: draft` on the strength of that evidence.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with content/pipelines/galaxy-workflow-review/scenarios.md and content/prompts/iwc/workflow-pr-review-command/index.md §Provenance and acceptance status, then check galaxyproject/iwc#1366 and run check:vendored against the local upstream checkouts. The work is done when the accepted prompt is re-vendored, the superset finding is recorded, a real PR exercises all phases and harness gates, and the evidence supports moving both artifacts out of draft.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, github
Domain
devtools, documentation, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.