nextflow-io / nextflow-io/nextflow
Directories fail to publish via `>>` operator in cloud environments
Nobody has claimed this yet.
- Dominant language
- Groovy
- Stars
- 3.5k
- Forks
- 811
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 61
Description
Environment
Nextflow version: 26.04.6 (build 12646)
Java version: openjdk version "21.0.7" 2025-04-15 LTS
Operating system: Tested with macOS and Debian
Description
Publishing directories (e.g. path('somedir/')) via the >> operator (e.g. path { dir -> dir >> "somewhere_else/${dir.name}/" }) silently fails when running on cloud environments (in my case Google Cloud).
Individual files publish fine, e.g. path('*.txt').
The below sections are vibe-coded/tested. I made some manual edits to reduce slop.
Minimal reproduction
main.nf:
nextflow.enable.dsl=2
process MAKE_DIR {
output:
path('outdir/'), topic: my_dir
script:
"""
mkdir -p outdir
echo hello > outdir/hello.txt
echo world > outdir/world.txt
"""
}
workflow {
main:
MAKE_DIR()
publish:
results = channel.topic('my_dir')
}
output {
results {
path { dir -> dir >> "published/${dir.name}/" }
}
}
// Note, this works:
// output {
// results {
// path 'published'
// }
// }
Run with below example command. -work-dir should be cloud storage bucket. -output-dir can be a cloud storage bucket or local path.
nextflow run main.nf -work-dir gs://<bucket>/work -output-dir ./out
Expected outcome: ./out/published/outdir/hello.txt and ./out/published/outdir/world.txt exist
Actual outcome: ./out/published/ doesn't even get created. No error/warning in .nextflow.log or console
Workaround
The temporary solution I'm using is to publish the directory contents as individual files:
publish:
results = channel.topic('my_dir').flatMap { dir -> dir.listFiles() }
Suspected root cause
file >> filepathbuilds asaveAslookup map keyed by the file's path relative to its task dir (PublishDsl.publish0).- For a directory, that key keeps its trailing slash:
"outdir/". - Before copying,
PublishDirdedups all paths by round-tripping them throughPathTrie(tokenize on/, rebuild). - That round-trip silently drops trailing empty segments, so the directory path comes back as
"outdir"— no slash. PublishDirlooks up"outdir"in a map keyed"outdir/"→ no match →null→ silently skips publishing. No error, no warning, no directory created.- Confirmed not a filesystem-provider issue: reproduces identically for
gs://workDir → local--outdirandgs://workDir →gs://--outdir. - Only directory-shaped values are affected — individual files never carry a trailing slash.
- Confirmed scoped to the
>>operator form only: plainpath 'somedir'string directive has nosaveAsmap, no lookup, no bug — same directory publishes fine with it.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the issue with the provided main.nf, a gs:// work directory, and local or cloud output. Read PublishDsl.publish0 in modules/nextflow/src/main/groovy/nextflow/extension/PublishOp.groovy and the PathTrie implementation, then trace the PublishDir lookup; done means the directory and both files appear under published/outdir without warnings.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- gcp, groovy
- Domain
- cloud, tooling
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100