Cannot stage remote input files from URL with query using

Bug report

Expected behavior and actual behavior

When getting an URL with a query (<URL>?<query>) contained in a file using e.g. splitCsv() the files specified using the query should be staged, but they are not; everything after the question mark (signifying query start) is omitted, resulting in no staging of the specified file(s).

There are already related issues (see e.g. #4092, #3023, #2918 and the explanation by @pditommaso in https://github.com/nextflow-io/nextflow/issues/4092#issuecomment-1656736661), but deal with earlier bugs and the fromPath() channel factory itself and adding glob: false. Those are different from this issue, as far as I can tell.

Steps to reproduce the problem

Workflow file:

workflow {
    gene_sets = Channel
        .fromPath ( "manifest.csv" )
        .splitCsv ( )
    PROCESS ( gene_sets.collect() )
}

process PROCESS {

    publishDir "results", mode: "copy"

    input:
    path(gene_sets)

    output:
    path(gene_sets)

    script:
    """
    echo ${gene_sets}
    """
}

Manifest file:

http://www.gsea-msigdb.org/gsea/msigdb/download_file.jsp?filePath=/msigdb/release/2022.1.Hs/h.all.v2022.1.Hs.symbols.gmt
http://www.gsea-msigdb.org/gsea/msigdb/download_file.jsp?filePath=/msigdb/release/2022.1.Hs/c5.go.bp.v2022.1.Hs.symbols.gmt
http://www.gsea-msigdb.org/gsea/msigdb/download_file.jsp?filePath=/msigdb/release/2022.1.Hs/c2.cp.kegg.v2022.1.Hs.symbols.gmt

Program output

Output:

$ nextflow run main.nf
 nextflow run main.nf
N E X T F L O W  ~  version 23.04.4
Launching `main.nf` [curious_jang] DSL2 - revision: 8be54217c6
[-        ] process > PROCESS -
ERROR ~ Error executing process > 'PROCESS'

Caused by:
  Process `PROCESS` input file name collision -- There are multiple input files for each of the following file names: download_file.jsp

Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line

 -- Check '.nextflow.log' file for details

Log:

Oct-13 16:45:51.236 [main] DEBUG nextflow.cli.Launcher - $> nextflow run main.nf
Oct-13 16:45:51.281 [main] INFO  nextflow.cli.CmdRun - N E X T F L O W  ~  version 23.04.4
Oct-13 16:45:51.291 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/Users/erikfasterius/.nextflow/plugins; core-plugins: nf-amazon@1.16.2,nf-azure@1.0.1,nf-codecommit@0.1.4,nf-console@1.0.5,nf-ga4gh@1.0.5,nf-google@1.7.3-patch1,nf-tower@1.5.12,nf-wave@0.8.4
Oct-13 16:45:51.297 [main] INFO  org.pf4j.DefaultPluginStatusProvider - Enabled plugins: []
Oct-13 16:45:51.297 [main] INFO  org.pf4j.DefaultPluginStatusProvider - Disabled plugins: []
Oct-13 16:45:51.299 [main] INFO  org.pf4j.DefaultPluginManager - PF4J version 3.4.1 in 'deployment' mode
Oct-13 16:45:51.303 [main] INFO  org.pf4j.AbstractPluginManager - No plugins
Oct-13 16:45:51.336 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 by global default
Oct-13 16:45:51.343 [main] INFO  nextflow.cli.CmdRun - Launching `main.nf` [curious_jang] DSL2 - revision: 8be54217c6
Oct-13 16:45:51.344 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[]
Oct-13 16:45:51.344 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[]
Oct-13 16:45:51.348 [main] DEBUG nextflow.secret.LocalSecretsProvider - Secrets store: /Users/erikfasterius/.nextflow/secrets/store.json
Oct-13 16:45:51.351 [main] DEBUG nextflow.secret.SecretsLoader - Discovered secrets providers: [nextflow.secret.LocalSecretsProvider@6331250e] - activable => nextflow.secret.LocalSecretsProvider@6331250e
Oct-13 16:45:51.392 [main] DEBUG nextflow.Session - Session UUID: 88c9779f-b53a-41c6-a65e-42d50256501e
Oct-13 16:45:51.392 [main] DEBUG nextflow.Session - Run name: curious_jang
Oct-13 16:45:51.393 [main] DEBUG nextflow.Session - Executor pool size: 12
Oct-13 16:45:51.403 [main] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'FileTransfer' minSize=10; maxSize=36; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false
Oct-13 16:45:51.416 [main] DEBUG nextflow.cli.CmdRun -
  Version: 23.04.4 build 5881
  Created: 25-09-2023 15:34 UTC (17:34 CEST)
  System: Mac OS X 13.5.2
  Runtime: Groovy 3.0.16 on OpenJDK 64-Bit Server VM 17.0.8.1+0
  Encoding: UTF-8 (UTF-8)
  Process: 61343@erikfmbp.ad [192.168.0.65]
  CPUs: 12 - Mem: 64 GB (31.5 GB) - Swap: 0 (0)
Oct-13 16:45:51.423 [main] DEBUG nextflow.Session - Work-dir: /Users/erikfasterius/projects/5969-rifseq/nextflow-reproducible-example/work [Mac OS X]
Oct-13 16:45:51.423 [main] DEBUG nextflow.Session - Script base path does not exist or is not a directory: /Users/erikfasterius/projects/5969-rifseq/nextflow-reproducible-example/bin
Oct-13 16:45:51.427 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[]
Oct-13 16:45:51.431 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory
Oct-13 16:45:51.440 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.DefaultCacheFactory
Oct-13 16:45:51.444 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 13; maxThreads: 1000
Oct-13 16:45:51.470 [main] DEBUG nextflow.Session - Session start
Oct-13 16:45:51.779 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution
Oct-13 16:45:51.851 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: null
Oct-13 16:45:51.851 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Oct-13 16:45:51.856 [main] DEBUG nextflow.executor.Executor - [warm up] executor > local
Oct-13 16:45:51.858 [main] DEBUG n.processor.LocalPollingMonitor - Creating local task monitor for executor 'local' > cpus=12; memory=64 GB; capacity=12; pollInterval=100ms; dumpInterval=5m
Oct-13 16:45:51.894 [main] DEBUG nextflow.Session - Workflow process names [dsl2]: PROCESS
Oct-13 16:45:51.894 [main] DEBUG nextflow.Session - Igniting dataflow network (2)
Oct-13 16:45:51.897 [main] DEBUG nextflow.processor.TaskProcessor - Starting process > PROCESS
Oct-13 16:45:51.897 [main] DEBUG nextflow.script.ScriptRunner - > Awaiting termination
Oct-13 16:45:51.897 [main] DEBUG nextflow.Session - Session await
Oct-13 16:45:52.869 [Actor Thread 9] DEBUG nextflow.processor.TaskProcessor - Process PROCESS > collision check staging file names: [download_file.jsp:3]
Oct-13 16:45:52.876 [Actor Thread 9] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=PROCESS; work-dir=null
  error [nextflow.exception.ProcessUnrecoverableException]: Process `PROCESS` input file name collision -- There are multiple input files for each of the following file names: download_file.jsp
Oct-13 16:45:52.886 [Actor Thread 9] ERROR nextflow.processor.TaskProcessor - Error executing process > 'PROCESS'

Caused by:
  Process `PROCESS` input file name collision -- There are multiple input files for each of the following file names: download_file.jsp

Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line
Oct-13 16:45:52.889 [Actor Thread 9] DEBUG nextflow.Session - Session aborted -- Cause: Process `PROCESS` input file name collision -- There are multiple input files for each of the following file names: download_file.jsp
Oct-13 16:45:52.896 [Actor Thread 9] DEBUG nextflow.Session - The following nodes are still active:
[process] PROCESS
  status=ACTIVE
  port 0: (value) bound ; channel: gene_sets
  port 1: (cntrl) -     ; channel: $

Oct-13 16:45:52.897 [main] DEBUG nextflow.Session - Session await > all processes finished
Oct-13 16:45:52.897 [main] DEBUG nextflow.Session - Session await > all barriers passed
Oct-13 16:45:52.897 [Actor Thread 7] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=PROCESS; work-dir=null
  error [java.lang.InterruptedException]: java.lang.InterruptedException
Oct-13 16:45:52.900 [main] DEBUG nextflow.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=0; failedCount=0; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=0ms; failedDuration=0ms; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=0; peakCpus=0; peakMemory=0; ]
Oct-13 16:45:53.098 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done
Oct-13 16:45:53.109 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye

So, there are two separate problems that I can see. Firstly: the remote file path gets truncated at the beginning of the query, making http://www.gsea-msigdb.org/gsea/msigdb/download_file.jsp?filePath=/msigdb/release/2022.1.Hs/h.all.v2022.1.Hs.symbols.gmt into http://www.gsea-msigdb.org/gsea/msigdb/download_file.jsp. Nextflow thus attempts to get three download_file.jsp files, but errors when it finds the duplicate entries.

Secondly: if one removes all but one entry in manifest.csv the duplicate file error disappears, but the truncation remains - there is no file at the truncated URL address, which yields a mostly empty HTML in the resulting download_file.jsp file:

download_file_jsp:

<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
</body>
</html>

I've tried a few things, to no avail. Adding glob: false to the fromPath() statement does nothing, as reading the manifest doesn't have any glob patterns and it doesn't seem that the glob statement propagates down to the file-reading in splitCsv(). I might be missing something, but reading the documentation and the other issues didn't help me get a solution so far.

Environment

• Nextflow version: 23.04.4
• Java version: 17.0.8.1
• Operating system: MacOS
• Bash version: GNU bash, version 5.2.15(1)-release (aarch64-apple-darwin22.1.0)

Reproduce the failure using the main.nf workflow and manifest.csv, then trace how fromPath() and splitCsv() handle remote URLs containing query parameters. Done means the full URL query is preserved during staging and the listed files are downloaded without filename collisions.