nextflow-io / nextflow-io/nextflow

Recursion with multiple inputs hangs with `until()`

Open
#2,609 7 comments 0 reactions 1 assignee View on GitHub

@bentsherman is already working on this.

Since Jul 11, 2022.

bug lang/dataflow
Dominant language
Groovy
Stars
3.5k
Forks
811
Avg merge
2d 11h
Merged PRs (30d)
61

Description

Bug report

Recursion with until() works when the process/workflow has one input/output but hangs at the end when there are multiple inputs/outputs.

Steps to reproduce the problem

I have provided a few recursion examples that work, along with modified examples that have multiple inputs and don't work with until() (but do work with times()).

until.nf

nextflow.preview.recursion=true

params.data = "$baseDir/hello.txt"

process foo {
  input:
    path 'input.txt'
  output:
    path 'result.txt'
  script:
    """
    cat input.txt > result.txt
    echo "Task $task.index added this" >> result.txt
    """
}

workflow {
  foo
    .recurse(file(params.data))
    .until{ it-> it.size() > 100 }

  foo
    .out
    .view(it->it.text)
}

until2.nf

nextflow.preview.recursion=true

params.data = "$baseDir/hello.txt"

process foo {
  echo true
  input:
    val i
    path 'input.txt'
  output:
    val i
    path 'result.txt'
  script:
    """
    cat input.txt > result.txt
    echo "Task ${i} added this" >> result.txt

    cat result.txt
    stat -c '%s' result.txt

    # ${i = i + 1}
    """
}

workflow {
  foo
    .recurse(1, file(params.data))
    .until{ i, f -> f.size() > 100 }
}

counter.nf

nextflow.preview.recursion=true

process count_down {
  input:
    val v
  output:
    val v
  exec:
    v = v - 1
}

workflow {
    n = 10
    count_down.recurse(n).until({ v -> v == 0 })
    count_down.out[0].view()
}

counter2.nf

nextflow.preview.recursion=true

process count_down {
  input:
    val u
    val v
  output:
    val u
    val v
  exec:
    u = u + 1
    v = v - 1
}

workflow {
    n = 10
    // count_down.recurse(0, n).times(n)
    count_down.recurse(0, n).until({ u, v -> v == 0 })
    count_down.out[1].view()
}
Program output

The examples until.nf and counter.nf complete successfully. The alternate examples until2.nf and counter2.nf hang at the very end. If you replace the until() line with the times() line, they complete successfully.

Environment
  • Nextflow version: 21.12.1-edge
  • Java version: 1.8.0_292
  • Operating system: Linux
  • Bash version: 4.4.19(1)-release (x86_64-pc-linux-gnu)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.