nextflow-io / nextflow-io/nextflow
Hidden files are not unstaged when using process.scratch
Nobody has claimed this yet.
- Dominant language
- Groovy
- Stars
- 3.5k
- Forks
- 811
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 61
Description
Bug report
Hidden files are not unstaged when using process.scratch
Expected behavior and actual behavior
Expected behaviour: Hidden files are unstaged when using process.scratch.
Actual behaviour: Hidden files are missing when process.scratch is enabled.
Steps to reproduce the problem
main.nf:
#! /usr/bin/env nextflow
workflow {
MAKE_HIDDEN_FILES | view | CHECK_HIDDEN_FILES
}
process MAKE_HIDDEN_FILES {
output:
path ("*.tab*", hidden: true)
script:
"""
touch myfile.tab .myfile.tab.1 .myfile.tab.2
"""
}
process CHECK_HIDDEN_FILES {
debug true
input:
path tab_files
script:
"""
ls -laR
"""
}
nextflow.config
process.scratch = true
Program output
process.scratch = false:
$ nextflow run main.nf
N E X T F L O W ~ version 22.04.3
Launching `main.nf` [cranky_murdock] DSL2 - revision: 86fad0a2f6
executor > local (2)
[d2/fd8919] process > MAKE_HIDDEN_FILES [100%] 1 of 1 ✔
executor > local (2)
[d2/fd8919] process > MAKE_HIDDEN_FILES [100%] 1 of 1 ✔[a8/3edeb4] process > CHECK_HIDDEN_FILES [100%] 1 of 1 ✔[/domus/h1/mahesh/test_dir/nxf_sandbox/test_hidden_file_staging/work/d2/fd89193143793afbb30e9b7aece6d1/.myfile.tab.1, /domus/h1/mahesh/test_dir/nxf_sandbox/test_hidden_file_staging/work/d2/fd89193143793afbb30e9b7aece6d1/.myfile.tab.2, /domus/h1/mahesh/test_dir/nxf_sandbox/test_hidden_file_staging/work/d2/fd89193143793afbb30e9b7aece6d1/myfile.tab]
.:
total 24
drwxrwxr-x 2 mahesh mahesh 4096 Jun 22 15:18 .
drwxrwxr-x 3 mahesh mahesh 4096 Jun 22 15:18 ..
-rw-rw-r-- 1 mahesh mahesh 0 Jun 22 15:18 .command.begin
-rw-rw-r-- 1 mahesh mahesh 0 Jun 22 15:18 .command.err
-rw-rw-r-- 1 mahesh mahesh 0 Jun 22 15:18 .command.log
-rw-rw-r-- 1 mahesh mahesh 0 Jun 22 15:18 .command.out
-rw-rw-r-- 1 mahesh mahesh 3499 Jun 22 15:18 .command.run
-rw-rw-r-- 1 mahesh mahesh 24 Jun 22 15:18 .command.sh
lrwxrwxrwx 1 mahesh mahesh 112 Jun 22 15:18 myfile.tab -> /domus/h1/mahesh/test_dir/nxf_sandbox/test_hidden_file_staging/work/d2/fd89193143793afbb30e9b7aece6d1/myfile.tab
lrwxrwxrwx 1 mahesh mahesh 115 Jun 22 15:18 .myfile.tab.1 -> /domus/h1/mahesh/test_dir/nxf_sandbox/test_hidden_file_staging/work/d2/fd89193143793afbb30e9b7aece6d1/.myfile.tab.1
lrwxrwxrwx 1 mahesh mahesh 115 Jun 22 15:18 .myfile.tab.2 -> /domus/h1/mahesh/test_dir/nxf_sandbox/test_hidden_file_staging/work/d2/fd89193143793afbb30e9b7aece6d1/.myfile.tab.2
process.scratch = true:
$ nextflow run main.nf
N E X T F L O W ~ version 22.04.3
Launching `main.nf` [awesome_lorenz] DSL2 - revision: 86fad0a2f6
executor > local (2)
[03/873db6] process > MAKE_HIDDEN_FILES [100%] 1 of 1 ✔
[98/2b8d45] process > CHECK_HIDDEN_FILES [100%] 1 of 1 ✔
/domus/h1/mahesh/test_dir/nxf_sandbox/test_hidden_file_staging/work/03/873db607bc6988e20d1cfc7145ee88/myfile.tab
.:
total 53184
drwx------ 2 mahesh mahesh 4096 Jun 22 15:20 .
drwxrwxrwt 1 root root 54435840 Jun 22 15:20 ..
-rw-rw-r-- 1 mahesh mahesh 0 Jun 22 15:20 .command.err
-rw-rw-r-- 1 mahesh mahesh 0 Jun 22 15:20 .command.out
lrwxrwxrwx 1 mahesh mahesh 112 Jun 22 15:20 myfile.tab -> /domus/h1/mahesh/test_dir/nxf_sandbox/test_hidden_file_staging/work/03/873db607bc6988e20d1cfc7145ee88/myfile.tab
Environment
- Nextflow version: 22.04.3
- Java version: openjdk 11.0.9.1-internal 2020-11-04
- Operating system: Linux
- Bash version: GNU bash, version 4.2.46(2)-release (x86_64-redhat-linux-gnu)
Additional context
I think it's caused by the eval "ls -1d *.tab*" | sort | uniq in the .command.run file, when unstaging files.
The fix might be as simple as eval "ls -1d *.tab*" | sort | uniq -> eval "ls -1d {.,}*.tab*" | sort | uniq, but I'm not sure if there's any issue with the not matching any hidden files for the majority of the time.
I think it's this line: https://github.com/nextflow-io/nextflow/blob/2b6f70a8fa55b993fa48755f7a47ac9e1b584e48/modules/nextflow/src/main/groovy/nextflow/executor/SimpleFileCopyStrategy.groovy#L186
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the referenced line in modules/nextflow/src/main/groovy/nextflow/executor/SimpleFileCopyStrategy.groovy and reproduce the issue with the provided main.nf and nextflow.config. Compare staging with process.scratch disabled and enabled; done means hidden files such as .myfile.tab.1 and .myfile.tab.2 are available in CHECK_HIDDEN_FILES.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- bash, groovy
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100