nextflow-io / nextflow-io/nextflow
Profile config is applied/evaluated at the point of definition rather than after any standard config
Nobody has claimed this yet.
- Dominant language
- Groovy
- Stars
- 3.5k
- Forks
- 811
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 61
Description
Per discussion on #4960 (PR: #4744) any config defined in a profile is evaluated in the order it was defined in the files with regards to non-profile config:
profiles {
special {
process {
ext.args = "special"
cpus = 8
withLabel: example {
memory = 16.GB
}
}
}
}
process {
ext.args = ""
cpus = 2
withLabel: example {
memory = 4.GB
}
}
In this example applying -profile special will have no effect as the process config will be superseded by the top-level process block defined after it.
It seems that this is the expected behaviour but I think that it is not well described in docs and additionally confusing now that profiles are applied in the order they are passed on the CLI and not the order they are defined in files. The same is true if the profile code is included via an includeConfig statement.
It would seem more logical that any profile is applied after all other config, with profiles themselves applied in the given order from the CLI?
Further examples
I think this example is particularly counterintuitive.
Using -profile override2,override then override2 will be correctly applied but override will be masked by code defined between the includeConfig even though the override profile is supposed to be applied after override2.
includeConfig './nextflow2.config' // defines profile `override`
process {
withName: '(.*):?A' {
ext.args = 'a'
ext.args2 = 'a'
}
withName: '(.*):?B' {
ext.args = 'b'
ext.args2 = 'b'
}
}
includeConfig './nextflow3.config' // defines profile `override2`
Here using -profile override2,override the override profile is correctly applied even though it is defined before override2
profiles {
override {
process {
ext.args = "override"
cpus = 4
withLabel: example {
memory = 8.GB
}
}
}
override2 {
process {
ext.args = "override2"
cpus = 8
withLabel: example {
memory = 16.GB
}
}
}
}
Possible solutions
- apply/resolve any profile config after all other non-profile config is resolved
- document the current behaviour better and make clear that to avoid any issues all profiles should be included/defined as the final step of any config
Environment
- Nextflow version: 26.03.2
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the examples using profiles, includeConfig, and the CLI -profile order described in the issue. Trace how profile and non-profile configuration are resolved, then define whether the result should change or the current ordering should be documented and covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- groovy
- Domain
- tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100