nextflow-io / nextflow-io/nextflow

When files from different processess have the same name, workflow outputs only copy one file

Open
#6,617 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Groovy
Stars
3.5k
Forks
811
Avg merge
2d 11h
Merged PRs (30d)
61

Description

Bug report

Expected behavior and actual behavior

I have two processes that both create files with the same name.
I expect to be able to publish both files and update the names in the output block.

Instead, only the second file gets output. eg:

output {
    profiles {
        mode 'copy'
        path { out ->
            // A file named "pyspy.prof" from the PROFILE_A process
            out.profile_a >> "pyspy/${out.meta.id}.profile_a.pyspy.prof"
            // A file named "pyspy.prof" from the PROFILE_B process
            out.profile_b >> "pyspy/${out.meta.id}.profile_b.pyspy.prof"
}}}

I would expect to get outputs such as:

results
└── pyspy
    ├── case001.profile_a.pyspy.prof
    ├── case001.profile_b.pyspy.prof
    ├── case002.profile_a.pyspy.prof
    └── case002.profile_b.pyspy.prof

But what I actually get is:

results
└── pyspy
    ├── case001.profile_b.pyspy.prof
    └── case002.profile_b.pyspy.prof

Furthermore, the file content is incorrect:

head ./results/pyspy/*

==> ./results/pyspy/case001.profile_b.pyspy.prof <==
PROFILE_A

==> ./results/pyspy/case002.profile_b.pyspy.prof <==
PROFILE_B

I can confirm that the files were not mis-attributed in the join. The following is the output from the toy example below:

[meta:[id:case001], profile_a:/Users/gabrielaldam/Projects/consolidate_abtiering/fail/work/87/0de5c7ec9482e8f25a0f8d0e553bc6/pyspy.prof, profile_b:/Users/gabrielaldam/Projects/consolidate_abtiering/fail/work/89/ad6c7974848a11b1fba224b09f0023/pyspy.prof]
[meta:[id:case002], profile_a:/Users/gabrielaldam/Projects/consolidate_abtiering/fail/work/56/5827cde5334bde58db0053e8300bc8/pyspy.prof, profile_b:/Users/gabrielaldam/Projects/consolidate_abtiering/fail/work/df/012645b58f7fe7a0b7ca189b6f02ed/pyspy.prof]

And the file content:

cat /Users/gabrielaldam/Projects/consolidate_abtiering/fail/work/87/0de5c7ec9482e8f25a0f8d0e553bc6/pyspy.prof
PROFILE_A
cat /Users/gabrielaldam/Projects/consolidate_abtiering/fail/work/89/ad6c7974848a11b1fba224b09f0023/pyspy.prof
PROFILE_B
cat /Users/gabrielaldam/Projects/consolidate_abtiering/fail/work/56/5827cde5334bde58db0053e8300bc8/pyspy.prof
PROFILE_A
cat /Users/gabrielaldam/Projects/consolidate_abtiering/fail/work/df/012645b58f7fe7a0b7ca189b6f02ed/pyspy.prof
PROFILE_B
Steps to reproduce the problem

The following example can reproduce the issue:

workflow {
    main:
    ch_cases = channel.of('case001','case002')
    PROFILE_A(ch_cases)
    PROFILE_B(ch_cases)

    ch_profiles = PROFILE_A.out.profiles.join(PROFILE_B.out.profiles)
        .map { caseid, profile_a, profile_b ->
            [
                meta:[ id: caseid ],
                profile_a: profile_a,
                profile_b: profile_b
            ]
        }
    ch_profiles.view()


    publish:
    profiles = ch_profiles
}

output {
    profiles {
        mode 'copy'
        path { out ->
            out.profile_a >> "pyspy/${out.meta.id}.profile_a.pyspy.prof"
            out.profile_b >> "pyspy/${out.meta.id}.profile_b.pyspy.prof"
        }
    }
}

process PROFILE_A {
    input:
    val caseid

    script:
    """
    echo "PROFILE_A" > pyspy.prof
    """

    output:
    tuple val(caseid), path("pyspy.prof"), emit: profiles
}

process PROFILE_B {
    input:
    val caseid

    script:
    """
    echo "PROFILE_B" > pyspy.prof
    """

    output:
    tuple val(caseid), path("pyspy.prof"), emit: profiles
}
Program output
Nov-27 15:44:12.308 [main] DEBUG nextflow.cli.Launcher - $> nextflow run ./main.nf
Nov-27 15:44:12.356 [main] DEBUG nextflow.cli.CmdRun - N E X T F L O W  ~  version 25.10.0
Nov-27 15:44:12.410 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/Users/gabrielaldam/.nextflow/plugins; core-plugins: nf-amazon@3.4.1,nf-azure@1.20.2,nf-cloudcache@0.5.0,nf-codecommit@0.5.0,nf-console@1.3.0,nf-google@1.23.3,nf-k8s@1.2.2,nf-tower@1.17.1,nf-wave@1.16.1
Nov-27 15:44:12.425 [main] INFO  o.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Nov-27 15:44:12.426 [main] INFO  o.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Nov-27 15:44:12.427 [main] INFO  org.pf4j.DefaultPluginManager - PF4J version 3.12.0 in 'deployment' mode
Nov-27 15:44:12.436 [main] DEBUG nextflow.util.RetryConfig - Missing nextflow session - using default retry config
Nov-27 15:44:12.534 [main] INFO  org.pf4j.AbstractPluginManager - No plugins
Nov-27 15:44:12.535 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[]
Nov-27 15:44:12.535 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[]
Nov-27 15:44:12.545 [main] DEBUG n.secret.LocalSecretsProvider - Secrets store: /Users/gabrielaldam/.nextflow/secrets/store.json
Nov-27 15:44:12.550 [main] DEBUG nextflow.secret.SecretsLoader - Discovered secrets providers: [nextflow.secret.LocalSecretsProvider@67c277a0] - activable => nextflow.secret.LocalSecretsProvider@67c277a0
Nov-27 15:44:12.572 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 by global default
Nov-27 15:44:12.588 [main] DEBUG nextflow.cli.CmdRun - Launching `./main.nf` [festering_mahavira] DSL2 - revision: d5c3127ffd
Nov-27 15:44:12.619 [main] DEBUG nextflow.Session - Session UUID: d0d03db4-48d4-48a0-8f82-5cc3c25072ee
Nov-27 15:44:12.619 [main] DEBUG nextflow.Session - Run name: festering_mahavira
Nov-27 15:44:12.619 [main] DEBUG nextflow.Session - Executor pool size: 8
Nov-27 15:44:12.624 [main] DEBUG nextflow.file.FilePorter - File porter settings maxRetries=3; maxTransfers=50; pollTimeout=null
Nov-27 15:44:12.627 [main] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'FileTransfer' minSize=10; maxSize=24; workQueue=LinkedBlockingQueue[-1]; allowCoreThreadTimeout=false
Nov-27 15:44:12.648 [main] DEBUG nextflow.cli.CmdRun -
  Version: 25.10.0 build 10289
  Created: 22-10-2025 16:26 UTC (17:26 BST)
  System: Mac OS X 26.1
  Runtime: Groovy 4.0.28 on OpenJDK 64-Bit Server VM 17.0.15+6-LTS
  Encoding: UTF-8 (UTF-8)
  Process: 81021@GEL-G9C747KD9H [10.255.51.115]
  CPUs: 8 - Mem: 16 GB (76 MB) - Swap: 6 GB (1.1 GB)
Nov-27 15:44:12.657 [main] DEBUG nextflow.Session - Work-dir: /Users/gabrielaldam/Projects/consolidate_abtiering/fail/work [Mac OS X]
Nov-27 15:44:12.657 [main] DEBUG nextflow.Session - Script base path does not exist or is not a directory: /Users/gabrielaldam/Projects/consolidate_abtiering/fail/bin
Nov-27 15:44:12.662 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[]
Nov-27 15:44:12.666 [main] DEBUG nextflow.Session - Observer factory (v2): LinObserverFactory
Nov-27 15:44:12.667 [main] DEBUG nextflow.Session - Observer factory (v2): DefaultObserverFactory
Nov-27 15:44:12.686 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.DefaultCacheFactory
Nov-27 15:44:12.691 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 9; maxThreads: 1000
Nov-27 15:44:12.721 [main] DEBUG nextflow.Session - Session start
Nov-27 15:44:13.042 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Nov-27 15:44:13.104 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Nov-27 15:44:13.105 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Nov-27 15:44:13.109 [main] DEBUG nextflow.executor.Executor - [warm up] executor > local
Nov-27 15:44:13.112 [main] DEBUG n.processor.LocalPollingMonitor - Creating local task monitor for executor 'local' > cpus=8; memory=16 GB; capacity=8; pollInterval=100ms; dumpInterval=5m
Nov-27 15:44:13.113 [main] DEBUG n.processor.TaskPollingMonitor - >>> barrier register (monitor: local)
Nov-27 15:44:13.128 [main] DEBUG nextflow.processor.TaskProcessor - Creating process 'PROFILE_A': maxForks=0; fair=false; array=0
Nov-27 15:44:13.148 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Nov-27 15:44:13.148 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Nov-27 15:44:13.149 [main] DEBUG nextflow.processor.TaskProcessor - Creating process 'PROFILE_B': maxForks=0; fair=false; array=0
Nov-27 15:44:13.169 [main] DEBUG nextflow.Session - Workflow process names [dsl2]: PROFILE_A, PROFILE_B
Nov-27 15:44:13.170 [main] DEBUG nextflow.Session - Igniting dataflow network (3)
Nov-27 15:44:13.170 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > PROFILE_A
Nov-27 15:44:13.170 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > PROFILE_B
Nov-27 15:44:13.171 [main] DEBUG nextflow.script.ScriptRunner - Parsed script files:
  Script_50d6b2ddf702c0dd: /Users/gabrielaldam/Projects/consolidate_abtiering/fail/main.nf
Nov-27 15:44:13.171 [main] DEBUG nextflow.script.ScriptRunner - > Awaiting termination
Nov-27 15:44:13.171 [main] DEBUG nextflow.Session - Session await
Nov-27 15:44:13.246 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Nov-27 15:44:13.247 [Task submitter] INFO  nextflow.Session - [87/0de5c7] Submitted process > PROFILE_A (1)
Nov-27 15:44:13.252 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Nov-27 15:44:13.252 [Task submitter] INFO  nextflow.Session - [df/012645] Submitted process > PROFILE_B (2)
Nov-27 15:44:13.256 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Nov-27 15:44:13.256 [Task submitter] INFO  nextflow.Session - [89/ad6c79] Submitted process > PROFILE_B (1)
Nov-27 15:44:13.260 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Nov-27 15:44:13.260 [Task submitter] INFO  nextflow.Session - [56/5827cd] Submitted process > PROFILE_A (2)
Nov-27 15:44:13.323 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 1; name: PROFILE_A (1); status: COMPLETED; exit: 0; error: -; workDir: /Users/gabrielaldam/Projects/consolidate_abtiering/fail/work/87/0de5c7ec9482e8f25a0f8d0e553bc6]
Nov-27 15:44:13.324 [Task monitor] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'TaskFinalizer' minSize=10; maxSize=24; workQueue=LinkedBlockingQueue[-1]; allowCoreThreadTimeout=false
Nov-27 15:44:13.325 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 3; name: PROFILE_B (2); status: COMPLETED; exit: 0; error: -; workDir: /Users/gabrielaldam/Projects/consolidate_abtiering/fail/work/df/012645b58f7fe7a0b7ca189b6f02ed]
Nov-27 15:44:13.326 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 2; name: PROFILE_B (1); status: COMPLETED; exit: 0; error: -; workDir: /Users/gabrielaldam/Projects/consolidate_abtiering/fail/work/89/ad6c7974848a11b1fba224b09f0023]
Nov-27 15:44:13.326 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 4; name: PROFILE_A (2); status: COMPLETED; exit: 0; error: -; workDir: /Users/gabrielaldam/Projects/consolidate_abtiering/fail/work/56/5827cde5334bde58db0053e8300bc8]
Nov-27 15:44:13.360 [main] DEBUG nextflow.Session - Session await > all processes finished
Nov-27 15:44:13.373 [Actor Thread 6] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'PublishDir' minSize=10; maxSize=24; workQueue=LinkedBlockingQueue[-1]; allowCoreThreadTimeout=false
Nov-27 15:44:13.419 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: local) - terminating tasks monitor poll loop
Nov-27 15:44:13.419 [main] DEBUG nextflow.Session - Session await > all barriers passed
Nov-27 15:44:13.421 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'TaskFinalizer' shutdown completed (hard=false)
Nov-27 15:44:13.421 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'PublishDir' shutdown completed (hard=false)
Nov-27 15:44:13.424 [main] DEBUG n.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=4; failedCount=0; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=6ms; failedDuration=0ms; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=4; peakCpus=4; peakMemory=0; ]
Nov-27 15:44:13.627 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done
Nov-27 15:44:13.635 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'FileTransfer' shutdown completed (hard=false)
Nov-27 15:44:13.635 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye
Environment
  • Nextflow version: nextflow version 25.10.0.10289
  • Java version: OpenJDK 64-Bit Server VM Homebrew (build 22, mixed mode, sharing)
  • Operating system: macOS (26.1 (25B78))
  • Bash version: zsh 5.9 (arm64-apple-darwin25.0)
Additional context

(Add any other context about the problem here)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the supplied main.nf reproduction and inspect the output block's publish path handling for the joined profile_a and profile_b files. Trace how duplicate source filenames are tracked during publishing. Done means both files from each case are copied to their distinct renamed paths with their original contents preserved.

Written by the indexing model from the issue text.

Assessment

Tech stack
groovy
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.