nf-core / nf-core/nft-utils

Stricter handling of nf-test outputs

Open
#92 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
13
Forks
11
Avg merge
1d 16h
Merged PRs (30d)
6

Description

There are silent mishaps (i.e. non-failing test) in the nf-test output in nf-core/ampliseq that seem relevant for a pipeline to have fixed. But due to the test passing, those can be easily overlooked.

(1) in PR 1062, test 8/20 = '-profile test_multidb'
      > -[nf-core/ampliseq] Pipeline completed successfully-
      PASSED (785.039s)
    Snapshots:
      Obsolete snapshots can only be checked if all tests of a file are executed successful.

-> it seems to me that { assert !path("$outputDir/qiime2/barplot/level-8.csv").exists() }, refers to a not existing file leading to that message. But that seems important enough to me to report the test as failing.
Edit: As below might be a different problem (if the AI is correct).

(2) in e.g. (not the origin, because I didnt find it earlier) PR 1065, test 5/20 = '-profile test_glosed':
      > -[nf-core/ampliseq] Pipeline completed successfully-
  Error reading YAML file: /home/runner/_work/ampliseq/ampliseq/~/tests/a29655bff5567c226ec88c6ed9d3de74/output/pipeline_info/software_versions.yml (No such file or directory)
      PASSED (306.768s)

-> It appears that indeed that file isnt produced by the pipeline. I think that should result in a failing test.

I am suggesting to be stricter and make sure that those tests fail instead of pass. As a pipeline developer, I would like to catch theses issues without the need to dig into every log but instead trust the green check mark.

p.s.: Thanks for the useful tools!

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the referenced nf-test cases from PR 1062 and PR 1065 and inspect how their output is interpreted. Compare the missing snapshot and software_versions.yml messages with the reported green results; done means these missing or invalid outputs cause the relevant tests to fail visibly.

Written by the indexing model from the issue text.

Assessment

Domain
ci-cd, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.