galaxyproject / galaxyproject/planemo
workflow_lint --iwc should error, not warn, when a workflow has no test cases
- Dominant language
- Python
- Stars
- 110
- Forks
- 102
- Avg merge
- 4d 21h
- Merged PRs (30d)
- 13
Description
> **Posted by Claude (AI assistant) on behalf of jmchilton — they did not author this text personally.**
Under `--iwc`, `workflow_lint` treats a workflow with no discoverable test cases as a warning, so it passes at `--fail_level error`. The surrounding checks in the same profile treat weaker conditions as errors, which makes the outcome inconsistent.
What the IWC profile currently does:
| Condition | Severity |
|---|---|
| `.dockstore.yml` workflow entry missing the `testParameterFiles` key | **error** |
| `.dockstore.yml` references a test file that is absent from disk | **error** |
| `README.md` / `CHANGELOG.md` missing | **error** |
| Workflow has **no test cases at all** | warning |
So the profile errors when tests are not *declared*, but only warns when the workflow is not actually *tested*. The condition that matters most is the one that does not fail the lint.
### Reproduce
A workflow directory that satisfies every error-level check — `README.md`, `CHANGELOG.md`, `.dockstore.yml` with `testParameterFiles` pointing at a file that exists on disk — but whose tests planemo cannot discover:
```console
$ ls
CHANGELOG.md README.md example-tests.yml example.gxwf.yml
$ grep -A1 testParameterFiles .dockstore.yml
testParameterFiles:
- /example-tests.yml
$ planemo workflow_lint --report_level all --fail_level error --iwc .
.. WARNING: Workflow missing test cases.
$ echo $?
0
```
The test file is present and declared, so the absent-file check does not fire. `_tests_path` derives the expected name with `os.path.splitext`, so for `example.gxwf.yml` it looks for `example.gxwf-tests.yml` and finds nothing. The result is a green lint on a workflow that is never tested.
The gxformat2 double extension is what makes this easy to hit, but it is not the only way in — anything that leaves `cases(runnable)` empty produces the same warning-only outcome.
### Suggested change
Under `iwc_grade`, raise "Workflow missing test cases" from a warning to an error. `_lint_tsts` already has the lint context, so this could key off `lint_args["iwc_grade"]` the way `lint_release` and the dockstore best-practice checks already do.
The non-IWC default should presumably stay a warning — plenty of legitimate workflows outside IWC have no tests.
### Why this is IWC's documented expectation
The IWC contribution guidelines list test files among the required contents of a workflow repository, alongside `README.md` and `CHANGELOG.md`, which the profile already enforces as errors:
> workflow repository directories contain:
> * at least one `.ga` workflow file […]
> * as many [Planemo test file] as workflow files […]
> * a `README.md` and a `CHANGELOG.md` file.
— https://github.com/galaxyproject/iwc/blob/main/workflows/README.md
Since `--iwc` exists to encode that policy, an untested workflow arguably should not pass it.
### Context
Found while adding gxformat2 support to `galaxyproject/iwc-lab`, a sandbox mirror of the IWC deploy pipeline. A Format-2 workflow with a `-tests.yml` named on the `.ga` convention linted clean and would have deployed untested.
Related: #1692, which covers a separate `.ga`-specific assumption in `dockstore_init`.
Reproduced with planemo 0.75.47 on Python 3.11.
Contributor guide
Research direction
Start with the _lint_tsts entry point and inspect how lint_args["iwc_grade"] is handled by lint_release and the Dockstore best-practice checks. Reproduce the command from the issue with an undiscoverable test plan, then verify that IWC reports an error while the non-IWC default remains a warning.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli, testing-qa
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100