nextflow-io / nextflow-io/nextflow

Hidden files are not unstaged when using process.scratch

Open
#2,983 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

lang/processes triage/investigate
Dominant language
Groovy
Stars
3.5k
Forks
811
Avg merge
2d 11h
Merged PRs (30d)
61

Description

Bug report

Hidden files are not unstaged when using process.scratch

Expected behavior and actual behavior

Expected behaviour: Hidden files are unstaged when using process.scratch.
Actual behaviour: Hidden files are missing when process.scratch is enabled.

Steps to reproduce the problem

main.nf:

#! /usr/bin/env nextflow

workflow {
    MAKE_HIDDEN_FILES | view | CHECK_HIDDEN_FILES
}

process MAKE_HIDDEN_FILES {
    output:
    path ("*.tab*", hidden: true)

    script:
    """
    touch myfile.tab .myfile.tab.1 .myfile.tab.2
    """
}

process CHECK_HIDDEN_FILES {
    debug true

    input:
    path tab_files

    script:
    """
    ls -laR 
    """
}

nextflow.config

process.scratch = true
Program output

process.scratch = false:

$ nextflow run main.nf 
N E X T F L O W  ~  version 22.04.3
Launching `main.nf` [cranky_murdock] DSL2 - revision: 86fad0a2f6
executor >  local (2)
[d2/fd8919] process > MAKE_HIDDEN_FILES  [100%] 1 of 1 ✔
executor >  local (2)
[d2/fd8919] process > MAKE_HIDDEN_FILES  [100%] 1 of 1 ✔[a8/3edeb4] process > CHECK_HIDDEN_FILES [100%] 1 of 1 ✔[/domus/h1/mahesh/test_dir/nxf_sandbox/test_hidden_file_staging/work/d2/fd89193143793afbb30e9b7aece6d1/.myfile.tab.1, /domus/h1/mahesh/test_dir/nxf_sandbox/test_hidden_file_staging/work/d2/fd89193143793afbb30e9b7aece6d1/.myfile.tab.2, /domus/h1/mahesh/test_dir/nxf_sandbox/test_hidden_file_staging/work/d2/fd89193143793afbb30e9b7aece6d1/myfile.tab]
.:
total 24
drwxrwxr-x 2 mahesh mahesh 4096 Jun 22 15:18 .
drwxrwxr-x 3 mahesh mahesh 4096 Jun 22 15:18 ..
-rw-rw-r-- 1 mahesh mahesh    0 Jun 22 15:18 .command.begin
-rw-rw-r-- 1 mahesh mahesh    0 Jun 22 15:18 .command.err
-rw-rw-r-- 1 mahesh mahesh    0 Jun 22 15:18 .command.log
-rw-rw-r-- 1 mahesh mahesh    0 Jun 22 15:18 .command.out
-rw-rw-r-- 1 mahesh mahesh 3499 Jun 22 15:18 .command.run
-rw-rw-r-- 1 mahesh mahesh   24 Jun 22 15:18 .command.sh
lrwxrwxrwx 1 mahesh mahesh  112 Jun 22 15:18 myfile.tab -> /domus/h1/mahesh/test_dir/nxf_sandbox/test_hidden_file_staging/work/d2/fd89193143793afbb30e9b7aece6d1/myfile.tab
lrwxrwxrwx 1 mahesh mahesh  115 Jun 22 15:18 .myfile.tab.1 -> /domus/h1/mahesh/test_dir/nxf_sandbox/test_hidden_file_staging/work/d2/fd89193143793afbb30e9b7aece6d1/.myfile.tab.1
lrwxrwxrwx 1 mahesh mahesh  115 Jun 22 15:18 .myfile.tab.2 -> /domus/h1/mahesh/test_dir/nxf_sandbox/test_hidden_file_staging/work/d2/fd89193143793afbb30e9b7aece6d1/.myfile.tab.2

process.scratch = true:

$ nextflow run main.nf 
N E X T F L O W  ~  version 22.04.3
Launching `main.nf` [awesome_lorenz] DSL2 - revision: 86fad0a2f6
executor >  local (2)
[03/873db6] process > MAKE_HIDDEN_FILES  [100%] 1 of 1 ✔
[98/2b8d45] process > CHECK_HIDDEN_FILES [100%] 1 of 1 ✔
/domus/h1/mahesh/test_dir/nxf_sandbox/test_hidden_file_staging/work/03/873db607bc6988e20d1cfc7145ee88/myfile.tab
.:
total 53184
drwx------ 2 mahesh mahesh     4096 Jun 22 15:20 .
drwxrwxrwt 1 root   root   54435840 Jun 22 15:20 ..
-rw-rw-r-- 1 mahesh mahesh        0 Jun 22 15:20 .command.err
-rw-rw-r-- 1 mahesh mahesh        0 Jun 22 15:20 .command.out
lrwxrwxrwx 1 mahesh mahesh      112 Jun 22 15:20 myfile.tab -> /domus/h1/mahesh/test_dir/nxf_sandbox/test_hidden_file_staging/work/03/873db607bc6988e20d1cfc7145ee88/myfile.tab

Environment
  • Nextflow version: 22.04.3
  • Java version: openjdk 11.0.9.1-internal 2020-11-04
  • Operating system: Linux
  • Bash version: GNU bash, version 4.2.46(2)-release (x86_64-redhat-linux-gnu)
Additional context

I think it's caused by the eval "ls -1d *.tab*" | sort | uniq in the .command.run file, when unstaging files.

The fix might be as simple as eval "ls -1d *.tab*" | sort | uniq -> eval "ls -1d {.,}*.tab*" | sort | uniq, but I'm not sure if there's any issue with the not matching any hidden files for the majority of the time.

I think it's this line: https://github.com/nextflow-io/nextflow/blob/2b6f70a8fa55b993fa48755f7a47ac9e1b584e48/modules/nextflow/src/main/groovy/nextflow/executor/SimpleFileCopyStrategy.groovy#L186

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the referenced line in modules/nextflow/src/main/groovy/nextflow/executor/SimpleFileCopyStrategy.groovy and reproduce the issue with the provided main.nf and nextflow.config. Compare staging with process.scratch disabled and enabled; done means hidden files such as .myfile.tab.1 and .myfile.tab.2 are available in CHECK_HIDDEN_FILES.

Written by the indexing model from the issue text.

Assessment

Tech stack
bash, groovy
Domain
backend
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.