Stricter handling of nf-test outputs
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 13
- Forks
- 11
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 6
Description
There are silent mishaps (i.e. non-failing test) in the nf-test output in nf-core/ampliseq that seem relevant for a pipeline to have fixed. But due to the test passing, those can be easily overlooked.
(1) in PR 1062, test 8/20 = '-profile test_multidb'
> -[nf-core/ampliseq] Pipeline completed successfully-
PASSED (785.039s)
Snapshots:
Obsolete snapshots can only be checked if all tests of a file are executed successful.
-> it seems to me that { assert !path("$outputDir/qiime2/barplot/level-8.csv").exists() }, refers to a not existing file leading to that message. But that seems important enough to me to report the test as failing.
Edit: As below might be a different problem (if the AI is correct).
(2) in e.g. (not the origin, because I didnt find it earlier) PR 1065, test 5/20 = '-profile test_glosed':
> -[nf-core/ampliseq] Pipeline completed successfully-
Error reading YAML file: /home/runner/_work/ampliseq/ampliseq/~/tests/a29655bff5567c226ec88c6ed9d3de74/output/pipeline_info/software_versions.yml (No such file or directory)
PASSED (306.768s)
-> It appears that indeed that file isnt produced by the pipeline. I think that should result in a failing test.
I am suggesting to be stricter and make sure that those tests fail instead of pass. As a pipeline developer, I would like to catch theses issues without the need to dig into every log but instead trust the green check mark.
p.s.: Thanks for the useful tools!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the referenced nf-test cases from PR 1062 and PR 1065 and inspect how their output is interpreted. Compare the missing snapshot and software_versions.yml messages with the reported green results; done means these missing or invalid outputs cause the relevant tests to fail visibly.
Written by the indexing model from the issue text.
Assessment
- Domain
- ci-cd, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100