Inconsistent S3 support for command line filenames

Bug report

When invoking nextflow via the CLI, some files can be specified as an S3 URL. Others cannot. This is not always clear from the documentation.

This may turn into a feature request, depending on intended behavior.

Expected behavior and actual behavior

Using the command in "steps to reproduce" and replacing one path at a time, the following paths correctly work with S3. Thank you!

• Bucket dir
• Job trace file
• HTML report file
• params file

The following files do not work with S3:

• Log file
  • Does not throw an error, does not upload to s3 after job completes. Is only written to a local path: /home/ssm-user/s3:/mybucket/report/nextflow.log. (note that the s3:// was normalized to s3:/ as part of a filename)
  • Log file output does not indicate any ambiguity in deciding how to handle the path
• Job definition
  • Fails for an explicit filename: Not a valid project name: s3:/ (I'm ok with this one; most workflows consist of multiple files, and a little weird to fetch a workflow from one filename)
  • Fails for a path name (with and without trailing slash): Not a valid project name: s3://mybucket/workflow/ (given that an SCM folder is allowed, I'm surprised by this one)
  • In both cases, log file says Nextflow is Using SCM config path: /home/ssm-user/.nextflow/scm
• Config file
  • Execution fails with The specified configuration file does not exist: /home/ssm-user/s3:/mybucket/workflow/aws_batch.config.

Workarounds are possible for logs and job definition. But it might be useful to specify the config file separately, eg to reuse a workflow (from github) while submitting the Nextflow process as a container to AWS batch (per blog suggestion #2). One could imagine wanting to use a standard NF container for this, while specifying the custom config in a persistent location.

Steps to reproduce the problem

All behaviors reported below are based on the following command, which was run from within an AWS instance. (each file was toggled between S3 vs local). The command below represents the verified-working form of the workflow.

nextflow \
    -log nextflow.log \
  run aws_batch.nf \
    -with-trace s3://mybucket/report/trace.txt \
    -with-report s3://mybucket/report/report.html \
    -c aws_batch.config \
    -bucket-dir s3://mybucket/workdir/onerun

Environment

• Nextflow version: 23.4.4
• Java version: OpenJDK Runtime Environment Corretto-17.0.8.8.1 (build 17.0.8.1+8-LTS)
• Operating system: Amazon Linux
• Bash version: 5.2.15(1)-release (x86_64-amazon-linux-gnu)

Start with the reproduced Nextflow CLI command and test each path option: -log, run, -c, -with-trace, -with-report, and -bucket-dir. Compare how S3 URLs are handled for logs, job definitions, and config files versus the working paths. Done should establish consistent behavior or clearly document which filenames support S3, including the resulting errors or workarounds.