nextflow-io / nextflow-io/nextflow
When files from different processess have the same name, workflow outputs only copy one file
Nobody has claimed this yet.
- Dominant language
- Groovy
- Stars
- 3.5k
- Forks
- 811
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 61
Description
Bug report
Expected behavior and actual behavior
I have two processes that both create files with the same name.
I expect to be able to publish both files and update the names in the output block.
Instead, only the second file gets output. eg:
output {
profiles {
mode 'copy'
path { out ->
// A file named "pyspy.prof" from the PROFILE_A process
out.profile_a >> "pyspy/${out.meta.id}.profile_a.pyspy.prof"
// A file named "pyspy.prof" from the PROFILE_B process
out.profile_b >> "pyspy/${out.meta.id}.profile_b.pyspy.prof"
}}}
I would expect to get outputs such as:
results
└── pyspy
├── case001.profile_a.pyspy.prof
├── case001.profile_b.pyspy.prof
├── case002.profile_a.pyspy.prof
└── case002.profile_b.pyspy.prof
But what I actually get is:
results
└── pyspy
├── case001.profile_b.pyspy.prof
└── case002.profile_b.pyspy.prof
Furthermore, the file content is incorrect:
head ./results/pyspy/*
==> ./results/pyspy/case001.profile_b.pyspy.prof <==
PROFILE_A
==> ./results/pyspy/case002.profile_b.pyspy.prof <==
PROFILE_B
I can confirm that the files were not mis-attributed in the join. The following is the output from the toy example below:
[meta:[id:case001], profile_a:/Users/gabrielaldam/Projects/consolidate_abtiering/fail/work/87/0de5c7ec9482e8f25a0f8d0e553bc6/pyspy.prof, profile_b:/Users/gabrielaldam/Projects/consolidate_abtiering/fail/work/89/ad6c7974848a11b1fba224b09f0023/pyspy.prof]
[meta:[id:case002], profile_a:/Users/gabrielaldam/Projects/consolidate_abtiering/fail/work/56/5827cde5334bde58db0053e8300bc8/pyspy.prof, profile_b:/Users/gabrielaldam/Projects/consolidate_abtiering/fail/work/df/012645b58f7fe7a0b7ca189b6f02ed/pyspy.prof]
And the file content:
cat /Users/gabrielaldam/Projects/consolidate_abtiering/fail/work/87/0de5c7ec9482e8f25a0f8d0e553bc6/pyspy.prof
PROFILE_A
cat /Users/gabrielaldam/Projects/consolidate_abtiering/fail/work/89/ad6c7974848a11b1fba224b09f0023/pyspy.prof
PROFILE_B
cat /Users/gabrielaldam/Projects/consolidate_abtiering/fail/work/56/5827cde5334bde58db0053e8300bc8/pyspy.prof
PROFILE_A
cat /Users/gabrielaldam/Projects/consolidate_abtiering/fail/work/df/012645b58f7fe7a0b7ca189b6f02ed/pyspy.prof
PROFILE_B
Steps to reproduce the problem
The following example can reproduce the issue:
workflow {
main:
ch_cases = channel.of('case001','case002')
PROFILE_A(ch_cases)
PROFILE_B(ch_cases)
ch_profiles = PROFILE_A.out.profiles.join(PROFILE_B.out.profiles)
.map { caseid, profile_a, profile_b ->
[
meta:[ id: caseid ],
profile_a: profile_a,
profile_b: profile_b
]
}
ch_profiles.view()
publish:
profiles = ch_profiles
}
output {
profiles {
mode 'copy'
path { out ->
out.profile_a >> "pyspy/${out.meta.id}.profile_a.pyspy.prof"
out.profile_b >> "pyspy/${out.meta.id}.profile_b.pyspy.prof"
}
}
}
process PROFILE_A {
input:
val caseid
script:
"""
echo "PROFILE_A" > pyspy.prof
"""
output:
tuple val(caseid), path("pyspy.prof"), emit: profiles
}
process PROFILE_B {
input:
val caseid
script:
"""
echo "PROFILE_B" > pyspy.prof
"""
output:
tuple val(caseid), path("pyspy.prof"), emit: profiles
}
Program output
Nov-27 15:44:12.308 [main] DEBUG nextflow.cli.Launcher - $> nextflow run ./main.nf
Nov-27 15:44:12.356 [main] DEBUG nextflow.cli.CmdRun - N E X T F L O W ~ version 25.10.0
Nov-27 15:44:12.410 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/Users/gabrielaldam/.nextflow/plugins; core-plugins: nf-amazon@3.4.1,nf-azure@1.20.2,nf-cloudcache@0.5.0,nf-codecommit@0.5.0,nf-console@1.3.0,nf-google@1.23.3,nf-k8s@1.2.2,nf-tower@1.17.1,nf-wave@1.16.1
Nov-27 15:44:12.425 [main] INFO o.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Nov-27 15:44:12.426 [main] INFO o.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Nov-27 15:44:12.427 [main] INFO org.pf4j.DefaultPluginManager - PF4J version 3.12.0 in 'deployment' mode
Nov-27 15:44:12.436 [main] DEBUG nextflow.util.RetryConfig - Missing nextflow session - using default retry config
Nov-27 15:44:12.534 [main] INFO org.pf4j.AbstractPluginManager - No plugins
Nov-27 15:44:12.535 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[]
Nov-27 15:44:12.535 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[]
Nov-27 15:44:12.545 [main] DEBUG n.secret.LocalSecretsProvider - Secrets store: /Users/gabrielaldam/.nextflow/secrets/store.json
Nov-27 15:44:12.550 [main] DEBUG nextflow.secret.SecretsLoader - Discovered secrets providers: [nextflow.secret.LocalSecretsProvider@67c277a0] - activable => nextflow.secret.LocalSecretsProvider@67c277a0
Nov-27 15:44:12.572 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 by global default
Nov-27 15:44:12.588 [main] DEBUG nextflow.cli.CmdRun - Launching `./main.nf` [festering_mahavira] DSL2 - revision: d5c3127ffd
Nov-27 15:44:12.619 [main] DEBUG nextflow.Session - Session UUID: d0d03db4-48d4-48a0-8f82-5cc3c25072ee
Nov-27 15:44:12.619 [main] DEBUG nextflow.Session - Run name: festering_mahavira
Nov-27 15:44:12.619 [main] DEBUG nextflow.Session - Executor pool size: 8
Nov-27 15:44:12.624 [main] DEBUG nextflow.file.FilePorter - File porter settings maxRetries=3; maxTransfers=50; pollTimeout=null
Nov-27 15:44:12.627 [main] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'FileTransfer' minSize=10; maxSize=24; workQueue=LinkedBlockingQueue[-1]; allowCoreThreadTimeout=false
Nov-27 15:44:12.648 [main] DEBUG nextflow.cli.CmdRun -
Version: 25.10.0 build 10289
Created: 22-10-2025 16:26 UTC (17:26 BST)
System: Mac OS X 26.1
Runtime: Groovy 4.0.28 on OpenJDK 64-Bit Server VM 17.0.15+6-LTS
Encoding: UTF-8 (UTF-8)
Process: 81021@GEL-G9C747KD9H [10.255.51.115]
CPUs: 8 - Mem: 16 GB (76 MB) - Swap: 6 GB (1.1 GB)
Nov-27 15:44:12.657 [main] DEBUG nextflow.Session - Work-dir: /Users/gabrielaldam/Projects/consolidate_abtiering/fail/work [Mac OS X]
Nov-27 15:44:12.657 [main] DEBUG nextflow.Session - Script base path does not exist or is not a directory: /Users/gabrielaldam/Projects/consolidate_abtiering/fail/bin
Nov-27 15:44:12.662 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[]
Nov-27 15:44:12.666 [main] DEBUG nextflow.Session - Observer factory (v2): LinObserverFactory
Nov-27 15:44:12.667 [main] DEBUG nextflow.Session - Observer factory (v2): DefaultObserverFactory
Nov-27 15:44:12.686 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.DefaultCacheFactory
Nov-27 15:44:12.691 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 9; maxThreads: 1000
Nov-27 15:44:12.721 [main] DEBUG nextflow.Session - Session start
Nov-27 15:44:13.042 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Nov-27 15:44:13.104 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Nov-27 15:44:13.105 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Nov-27 15:44:13.109 [main] DEBUG nextflow.executor.Executor - [warm up] executor > local
Nov-27 15:44:13.112 [main] DEBUG n.processor.LocalPollingMonitor - Creating local task monitor for executor 'local' > cpus=8; memory=16 GB; capacity=8; pollInterval=100ms; dumpInterval=5m
Nov-27 15:44:13.113 [main] DEBUG n.processor.TaskPollingMonitor - >>> barrier register (monitor: local)
Nov-27 15:44:13.128 [main] DEBUG nextflow.processor.TaskProcessor - Creating process 'PROFILE_A': maxForks=0; fair=false; array=0
Nov-27 15:44:13.148 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Nov-27 15:44:13.148 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Nov-27 15:44:13.149 [main] DEBUG nextflow.processor.TaskProcessor - Creating process 'PROFILE_B': maxForks=0; fair=false; array=0
Nov-27 15:44:13.169 [main] DEBUG nextflow.Session - Workflow process names [dsl2]: PROFILE_A, PROFILE_B
Nov-27 15:44:13.170 [main] DEBUG nextflow.Session - Igniting dataflow network (3)
Nov-27 15:44:13.170 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > PROFILE_A
Nov-27 15:44:13.170 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > PROFILE_B
Nov-27 15:44:13.171 [main] DEBUG nextflow.script.ScriptRunner - Parsed script files:
Script_50d6b2ddf702c0dd: /Users/gabrielaldam/Projects/consolidate_abtiering/fail/main.nf
Nov-27 15:44:13.171 [main] DEBUG nextflow.script.ScriptRunner - > Awaiting termination
Nov-27 15:44:13.171 [main] DEBUG nextflow.Session - Session await
Nov-27 15:44:13.246 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Nov-27 15:44:13.247 [Task submitter] INFO nextflow.Session - [87/0de5c7] Submitted process > PROFILE_A (1)
Nov-27 15:44:13.252 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Nov-27 15:44:13.252 [Task submitter] INFO nextflow.Session - [df/012645] Submitted process > PROFILE_B (2)
Nov-27 15:44:13.256 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Nov-27 15:44:13.256 [Task submitter] INFO nextflow.Session - [89/ad6c79] Submitted process > PROFILE_B (1)
Nov-27 15:44:13.260 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run
Nov-27 15:44:13.260 [Task submitter] INFO nextflow.Session - [56/5827cd] Submitted process > PROFILE_A (2)
Nov-27 15:44:13.323 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 1; name: PROFILE_A (1); status: COMPLETED; exit: 0; error: -; workDir: /Users/gabrielaldam/Projects/consolidate_abtiering/fail/work/87/0de5c7ec9482e8f25a0f8d0e553bc6]
Nov-27 15:44:13.324 [Task monitor] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'TaskFinalizer' minSize=10; maxSize=24; workQueue=LinkedBlockingQueue[-1]; allowCoreThreadTimeout=false
Nov-27 15:44:13.325 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 3; name: PROFILE_B (2); status: COMPLETED; exit: 0; error: -; workDir: /Users/gabrielaldam/Projects/consolidate_abtiering/fail/work/df/012645b58f7fe7a0b7ca189b6f02ed]
Nov-27 15:44:13.326 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 2; name: PROFILE_B (1); status: COMPLETED; exit: 0; error: -; workDir: /Users/gabrielaldam/Projects/consolidate_abtiering/fail/work/89/ad6c7974848a11b1fba224b09f0023]
Nov-27 15:44:13.326 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 4; name: PROFILE_A (2); status: COMPLETED; exit: 0; error: -; workDir: /Users/gabrielaldam/Projects/consolidate_abtiering/fail/work/56/5827cde5334bde58db0053e8300bc8]
Nov-27 15:44:13.360 [main] DEBUG nextflow.Session - Session await > all processes finished
Nov-27 15:44:13.373 [Actor Thread 6] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'PublishDir' minSize=10; maxSize=24; workQueue=LinkedBlockingQueue[-1]; allowCoreThreadTimeout=false
Nov-27 15:44:13.419 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: local) - terminating tasks monitor poll loop
Nov-27 15:44:13.419 [main] DEBUG nextflow.Session - Session await > all barriers passed
Nov-27 15:44:13.421 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'TaskFinalizer' shutdown completed (hard=false)
Nov-27 15:44:13.421 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'PublishDir' shutdown completed (hard=false)
Nov-27 15:44:13.424 [main] DEBUG n.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=4; failedCount=0; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=6ms; failedDuration=0ms; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=4; peakCpus=4; peakMemory=0; ]
Nov-27 15:44:13.627 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done
Nov-27 15:44:13.635 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'FileTransfer' shutdown completed (hard=false)
Nov-27 15:44:13.635 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye
Environment
- Nextflow version: nextflow version 25.10.0.10289
- Java version: OpenJDK 64-Bit Server VM Homebrew (build 22, mixed mode, sharing)
- Operating system: macOS (26.1 (25B78))
- Bash version:
zsh 5.9 (arm64-apple-darwin25.0)
Additional context
(Add any other context about the problem here)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the supplied main.nf reproduction and inspect the output block's publish path handling for the joined profile_a and profile_b files. Trace how duplicate source filenames are tracked during publishing. Done means both files from each case are copied to their distinct renamed paths with their original contents preserved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- groovy
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100