galaxyproject / galaxyproject/planemo

Workflow test collection assertions are silently skipped unless spelled `element_tests`

Open
#1,705 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
110
Forks
102
Avg merge
4d 21h
Merged PRs (30d)
13

Description

🤖 This issue was written by Claude (AI assistant) on jmchilton's behalf, not authored by them personally.

### Summary

Planemo and Galaxy's own framework workflow test runner read the same workflow test YAML but use
different rules to decide that an output assertion describes a *collection*. Each silently ignores the
other's spelling — no error, no warning, the assertion is just skipped. Planemo additionally drops
`collection_type`, the only spelling the published schema accepts.

### Detection disagreement

Planemo, `planemo/test/_check_output.py`:

```python
def for_collections(test_properties):
return "element_tests" in test_properties
```

Galaxy, `lib/galaxy_test/workflow/test_framework_workflows.py`:

```python
is_collection_test = isinstance(test_properties, dict) and (
"elements" in test_properties or test_properties.get("class") == "Collection"
)
```

So an output written `class: Collection` + `elements:` — the spelling used throughout Galaxy's own
framework workflow tests — is not seen as a collection by Planemo. It falls through to
`_check_output_file` and fails with `No path specified for expected output file`, which gives no hint
that the collection assertion was the thing that went unread.

### `collection_type` is dropped

`_check_output_collection` calls `TestCollectionOutputDef.from_dict(test_properties)`. Galaxy ships a
second constructor, `from_yaml_test_format` (`lib/galaxy/tool_util/parser/interface.py`), written for
exactly this file format — it resolves the `elements` / `element_tests` alias and maps top-level
`collection_type` onto `attributes["type"]`:

```python
@staticmethod
def from_yaml_test_format(as_dict):
if "attributes" not in as_dict:
as_dict["attributes"] = {}
attributes = as_dict["attributes"]
if "elements" in as_dict or "element_tests" in as_dict:
as_dict["element_tests"] = resolve_element_tests(as_dict)
if "collection_type" in as_dict:
attributes["type"] = as_dict["collection_type"]
return TestCollectionOutputDef.from_dict(as_dict)
```

Because Planemo uses `from_dict` instead, top-level `collection_type` is discarded and the collection
type simply is not asserted. The only spelling Planemo honours at runtime is
`attributes: {type: ...}` — and that one is rejected by the published schema, since
`CollectionAttributes` in `galaxy-tool-util-models` is `extra="forbid"` and declares only
`collection_type`:

```
INVALID attributes: {type: list} + element_tests -> attributes.type: Extra inputs are not permitted
VALID class: Collection + collection_type + element_count + element_tests
```

That leaves no spelling that is both schema-valid and type-asserting under Planemo.

### Suggested fix

In `_check_output_collection`, call `TestCollectionOutputDef.from_yaml_test_format(test_properties)`
rather than `from_dict`, and widen `for_collections` to match Galaxy's gate:

```python
def for_collections(test_properties):
return (
"element_tests" in test_properties
or "elements" in test_properties
or test_properties.get("class") == "Collection"
)
```

### Impact

IWC carries 236 collection output assertions; 206 use `class: Collection` + `element_tests` and 30 use
bare `element_tests`. The paired spelling works today only because `element_tests` happens to be present
alongside `class: Collection` — the `class` key is doing nothing for Planemo. None of the 236 assert a
collection type, which is consistent with there being no spelling that both validates and works.

Found while adding workflow tests to galaxyproject/foundry; verified against planemo 0.75.47 and
galaxy-tool-util-models as published on PyPI.

---

Mirror-image bug filed upstream in Galaxy: https://github.com/galaxyproject/galaxy/issues/23544

Contributor guide

Open the contributing guide

Research direction

Start in planemo/test/_check_output.py, reading for_collections and _check_output_collection, then compare their behavior with lib/galaxy/tool_util/parser/interface.py. Update collection detection and parsing so schema-valid class: Collection, elements or element_tests, and collection_type assertions are recognized. Verify against the workflow collection assertions described in the issue, including the 236 IWC cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.