galaxyproject / galaxyproject/planemo

workflow_lint --iwc should error, not warn, when a workflow has no test cases

Open Beginner friendly
#1,693 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
110
Forks
102
Avg merge
4d 21h
Merged PRs (30d)
13

Description

> **Posted by Claude (AI assistant) on behalf of jmchilton — they did not author this text personally.**

Under `--iwc`, `workflow_lint` treats a workflow with no discoverable test cases as a warning, so it passes at `--fail_level error`. The surrounding checks in the same profile treat weaker conditions as errors, which makes the outcome inconsistent.

What the IWC profile currently does:

| Condition | Severity |
|---|---|
| `.dockstore.yml` workflow entry missing the `testParameterFiles` key | **error** |
| `.dockstore.yml` references a test file that is absent from disk | **error** |
| `README.md` / `CHANGELOG.md` missing | **error** |
| Workflow has **no test cases at all** | warning |

So the profile errors when tests are not *declared*, but only warns when the workflow is not actually *tested*. The condition that matters most is the one that does not fail the lint.

### Reproduce

A workflow directory that satisfies every error-level check — `README.md`, `CHANGELOG.md`, `.dockstore.yml` with `testParameterFiles` pointing at a file that exists on disk — but whose tests planemo cannot discover:

```console
$ ls
CHANGELOG.md README.md example-tests.yml example.gxwf.yml
$ grep -A1 testParameterFiles .dockstore.yml
testParameterFiles:
- /example-tests.yml
$ planemo workflow_lint --report_level all --fail_level error --iwc .
.. WARNING: Workflow missing test cases.
$ echo $?
0
```

The test file is present and declared, so the absent-file check does not fire. `_tests_path` derives the expected name with `os.path.splitext`, so for `example.gxwf.yml` it looks for `example.gxwf-tests.yml` and finds nothing. The result is a green lint on a workflow that is never tested.

The gxformat2 double extension is what makes this easy to hit, but it is not the only way in — anything that leaves `cases(runnable)` empty produces the same warning-only outcome.

### Suggested change

Under `iwc_grade`, raise "Workflow missing test cases" from a warning to an error. `_lint_tsts` already has the lint context, so this could key off `lint_args["iwc_grade"]` the way `lint_release` and the dockstore best-practice checks already do.

The non-IWC default should presumably stay a warning — plenty of legitimate workflows outside IWC have no tests.

### Why this is IWC's documented expectation

The IWC contribution guidelines list test files among the required contents of a workflow repository, alongside `README.md` and `CHANGELOG.md`, which the profile already enforces as errors:

> workflow repository directories contain:
> * at least one `.ga` workflow file […]
> * as many [Planemo test file] as workflow files […]
> * a `README.md` and a `CHANGELOG.md` file.

— https://github.com/galaxyproject/iwc/blob/main/workflows/README.md

Since `--iwc` exists to encode that policy, an untested workflow arguably should not pass it.

### Context

Found while adding gxformat2 support to `galaxyproject/iwc-lab`, a sandbox mirror of the IWC deploy pipeline. A Format-2 workflow with a `-tests.yml` named on the `.ga` convention linted clean and would have deployed untested.

Related: #1692, which covers a separate `.ga`-specific assumption in `dockstore_init`.

Reproduced with planemo 0.75.47 on Python 3.11.

Contributor guide

Open the contributing guide

Research direction

Start with the _lint_tsts entry point and inspect how lint_args["iwc_grade"] is handled by lint_release and the Dockstore best-practice checks. Reproduce the command from the issue with an undiscoverable test plan, then verify that IWC reports an error while the non-IWC default remains a warning.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, testing-qa
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.