nextflow-io / nextflow-io/nextflow

Error handling for processes

Open
#903 62 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

lang/processes
Dominant language
Groovy
Stars
3.5k
Forks
811
Avg merge
2d 11h
Merged PRs (30d)
61

Description

We would like NF to support metadata tracking and routing; this would for example enable a process to inform another process that an action was not taken or a stage not passed, this could trigger other tasks. I include two examples from our current pipeline below; perhaps not a perfect match, but they informed my initial thinking about this.

    process irods {
        tag "${samplename}"

        input:
            val samplename from sample_list.flatMap{ it.readLines() }

        output:
            set val(samplename), file('*.cram') optional true into ch_cram_files
            file('*.lostcause.txt') optional true into ch_lostcause_irods

        script:
        """
        if bash -euo pipefail irods.sh -t ${params.studyid} -s ${samplename}; then
          true
        else
          stat=\$?
          tag='UNKNOWN'
          if [[ \$stat == 64 ]]; then tag='nofiles'; fi
          echo -e "${samplename}\\tirods\\t\$tag" > ${samplename}.lostcause.txt
        fi
        """
   }

# later ...
    ch_star_accept = Channel.create()
    ch_star_reject = Channel.create()

    star_aligned
        .choice(ch_star_accept, ch_star_reject)
            { namelogsbams -> check_log(namelogsbams[1]) ? 0 : 1 }

    ch_star_accept
    .map    { name, logs, bams -> [name, bams] }
    .into   { ch_featurecounts; ch_indexbam }

    ch_star_reject
    .map    { it -> "${it[0]}\tSTAR\tlowmapping\n" }
    .mix(ch_lostcause_irods)
    .set    { ch_lostcause }

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue provides Groovy process and channel examples but names no files or tests. Start by tracing the existing process-failure and channel-routing behavior, then clarify the metadata model and acceptance cases with maintainers. Done would require an agreed design and tests covering process status propagation and routing.

Written by the indexing model from the issue text.

Assessment

Tech stack
bash, groovy
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.