nextflow-io / nextflow-io/nextflow

Directories fail to publish via `>>` operator in cloud environments

Open
#7,398 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

storage/gcp triage/investigate
Dominant language
Groovy
Stars
3.5k
Forks
811
Avg merge
2d 11h
Merged PRs (30d)
61

Description

Environment

Nextflow version: 26.04.6 (build 12646)
Java version: openjdk version "21.0.7" 2025-04-15 LTS
Operating system: Tested with macOS and Debian

Description

Publishing directories (e.g. path('somedir/')) via the >> operator (e.g. path { dir -> dir >> "somewhere_else/${dir.name}/" }) silently fails when running on cloud environments (in my case Google Cloud).

Individual files publish fine, e.g. path('*.txt').

The below sections are vibe-coded/tested. I made some manual edits to reduce slop.

Minimal reproduction

main.nf:

nextflow.enable.dsl=2

process MAKE_DIR {
    output:
    path('outdir/'), topic: my_dir

    script:
    """
    mkdir -p outdir
    echo hello > outdir/hello.txt
    echo world > outdir/world.txt
    """
}

workflow {
    main:
    MAKE_DIR()

    publish:
    results = channel.topic('my_dir')
}

output {
    results {
        path { dir -> dir >> "published/${dir.name}/" }
    }
}


// Note, this works:
// output {
//     results {
//         path 'published'
//     }
// }

Run with below example command. -work-dir should be cloud storage bucket. -output-dir can be a cloud storage bucket or local path.

nextflow run main.nf -work-dir gs://<bucket>/work -output-dir ./out

Expected outcome: ./out/published/outdir/hello.txt and ./out/published/outdir/world.txt exist
Actual outcome: ./out/published/ doesn't even get created. No error/warning in .nextflow.log or console

Workaround

The temporary solution I'm using is to publish the directory contents as individual files:

publish:
results = channel.topic('my_dir').flatMap { dir -> dir.listFiles() }
Suspected root cause
  • file >> filepath builds a saveAs lookup map keyed by the file's path relative to its task dir (PublishDsl.publish0).
  • For a directory, that key keeps its trailing slash: "outdir/".
  • Before copying, PublishDir dedups all paths by round-tripping them through PathTrie (tokenize on /, rebuild).
  • That round-trip silently drops trailing empty segments, so the directory path comes back as "outdir" — no slash.
  • PublishDir looks up "outdir" in a map keyed "outdir/" → no match → null → silently skips publishing. No error, no warning, no directory created.
  • Confirmed not a filesystem-provider issue: reproduces identically for gs:// workDir → local --outdir and gs:// workDir → gs:// --outdir.
  • Only directory-shaped values are affected — individual files never carry a trailing slash.
  • Confirmed scoped to the >> operator form only: plain path 'somedir' string directive has no saveAs map, no lookup, no bug — same directory publishes fine with it.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the issue with the provided main.nf, a gs:// work directory, and local or cloud output. Read PublishDsl.publish0 in modules/nextflow/src/main/groovy/nextflow/extension/PublishOp.groovy and the PathTrie implementation, then trace the PublishDir lookup; done means the directory and both files appear under published/outdir without warnings.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, groovy
Domain
cloud, tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.