nextflow-io / nextflow-io/nextflow

Allow custom configs `params` to be parsed before nextflow.config

Open
#2,662 13 comments 9 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug lang/config
Dominant language
Groovy
Stars
3.5k
Forks
811
Avg merge
2d 11h
Merged PRs (30d)
61

Description

New feature

I'm not sure whether to classify this as a feature request or bug.
The issue is that params configuration from custom configs (-c) appears to be read in after nextflow.config.
The docs describe that params provided in a custom config have a higher priority than those in the nextflow.config. This is true at the time of running the workflow, but not when parsing and evaluating the conditions in the Nextflow config.

For example:
main.nf:

#! /usr/bin/env nextflow

nextflow.enable.dsl = 2

workflow {
    FOO()
}

process FOO {

    output:
    path 'myfile.txt'

    script:
    """
    touch myfile.txt
    """
}

nextflow.config:

params.outdir = 'results'
process {
    withName: 'FOO' {
        publishDir = "$params.outdir/foo"
    }
}

If one runs:

$ nextflow run main.nf --outdir '/my/new/path'

or

$ cat params.yml
outdir : '/my/new/path'
$ nextflow run main.nf -params-file params.yml

Then the publishDir is correctly set.

However, if one uses

$ cat custom.config
params.outdir = '/my/new/path'
$ nextflow run main.nf -c custom.config

Then the publishDir is not correct, instead using the default value.

Usage scenario

Lot's of Nextflow users currently provide custom params via the -c option, not realizing it is not applying to configuration in the nextflow.config.

Suggest implementation

Perhaps read in any custom configs in the order described by the docs before and implement the configuration evaluation in such a way that if the keys are already present, don't override them.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the behavior with main.nf, nextflow.config, custom.config, and params.yml using the documented nextflow run commands. Start by tracing how -c custom.config and nextflow.config are evaluated and ordered. Done means params supplied through -c are available when nextflow.config conditions and values are evaluated, while the documented precedence remains intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
groovy
Domain
backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.