galaxyproject / galaxyproject/planemo

Allow marking workflow steps to fail in tests

Open
#1,273 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
110
Forks
102
Avg merge
4d 21h
Merged PRs (30d)
13

Description

without failing all of the tests.

From matrix:

> Delphine
The main problem actually is that some of my workflows can't fully run with smaller data, most of it works fine but there alway a plotting or QC tool that fails because an output is empty. Even for artificially generated data, the hic files are 40 GB each. Is that ok to link the url of these, or is there a way to ignore some paths? because I have tried downsampling 10 different way it never gets both complete and smalll
Marius
Hmm, but the tools are in one of the github repos ? So those test inputs are much much smaller, but I agree that this is difficult to walk backwards from
Delphine
yes for example for hifiasm the primary assembly leads to 2 contigs and the alternate is empty it's fine for most of the workflow but the ploting tool that is basically just matplot lib fails on the empty stats files
Marius
ok, so if we could ignore / expect for some jobs to fail, would that be an option ?
We could annotate that in the tests
Delphine
that would be great, and it makes sense if we have some key datasets to verify it runs that should be enough

I don't know how that should look like in the test definitions, maybe

```
xfail:
- 3
- 4
- a step label
```
?

Of course using numeric ids might be a bit of a problem if we add or remove steps, so step labels are preferred. Using tool ids will also not work if you have multiple steps with the same tool. Maybe we should start guaranteeing that numeric ids are always augmented, then that'd be less of a problem ...

Contributor guide

Open the contributing guide

Research direction

Start by locating the workflow test definitions and the step execution path that reports failures. Define how expected failures are identified, preferably by stable step labels, and verify that marked failures do not fail the whole test while unexpected failures still do.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.