nextflow-io / nextflow-io/nextflow

Support passing multiple values to a single input in nextflow module run

Open
#7,212 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Groovy
Stars
3.5k
Forks
811
Avg merge
2d 11h
Merged PRs (30d)
61

Description

New feature

When running nextflow module run, there is currently no ergonomic way to pass multiple values for a single input parameter (path, val). For path inputs the only working approach is a glob pattern, which is fragile and not always applicable. For val and other input types there is no workaround at all.

Use case

When using nextflow module run to test or run a module that accepts multiple files for a single input (e.g. path(csv, stageAs: 'inputs/csv*/*') in nf-core/csvtk/concat), users naturally reach for one of the following CLI forms:

Space-separated values:

nextflow module run nf-core/csvtk/concat \
    --csv sample_a.csv sample_b.csv \
    --in_format csv --out_format csv --outdir results

Repeated flag:

nextflow module run nf-core/csvtk/concat \
    --csv sample_a.csv \
    --csv sample_b.csv \
    --in_format csv --out_format csv --outdir results

Both forms run without error but silently pass only one file to the process — the concatenation produces a copy of the first file rather than a combined result. There is no warning to the user.

The only currently working approach is a glob pattern:

nextflow module run nf-core/csvtk/concat \
    --csv "/path/to/sample_*.csv" \
    --in_format csv --out_format csv --outdir results

This works because Nextflow resolves the glob into a list via files(), but it triggers a WARN: The file() function was called with a glob pattern that matched a collection of files -- use files() instead. warning, requires files to share a naming pattern, and fails entirely when files live in different directories.

Suggested implementation

  1. Support space-separated multi-value path parameters in nextflow module run: when a parameter is declared as path type in meta.yml and multiple space-separated values are provided, resolve each token as a file and pass them as a list.

  2. Support repeated flags for path parameters: --csv a.csv --csv b.csv should accumulate into a list rather than overwriting.

  3. Emit a warning when a path parameter resolves to a single file but the process input declares it accepts multiple files (e.g. via stageAs patterns like inputs/csv*/*), to surface the silent-truncation bug users currently experience.

Either (1) or (2) would resolve the main issue; (3) is a safety net for the current behaviour.

sample_a.csv
sample_b.csv

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the nextflow module run CLI input handling and the meta.yml declarations for path parameters. Reproduce the space-separated and repeated-flag examples, then trace how values are resolved and passed to the process. Done means supported multi-value input forms accumulate all files, with the requested warning behavior covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
groovy
Domain
cli
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.