nextflow-io / nextflow-io/nextflow

Add config source attribution (blame) to `nextflow config` command

Open
#6,894 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

stale
Dominant language
Groovy
Stars
3.5k
Forks
811
Avg merge
2d 11h
Merged PRs (30d)
61

Description

Summary

Add a new flag to the nextflow config command that shows the source/origin of each configuration value, similar to git blame. This would help users debug config resolution issues by showing where each value was loaded from.

Motivation

Debugging configuration resolution can be difficult, especially when configs come from multiple sources (pipeline config, user config, CLI params, Platform/Seqera configs, etc.). Currently there's no easy way to determine which config file a particular value came from or which source "won" when multiple configs define the same value.

Proposed Solution

Add a --blame (or similar) flag to nextflow config that annotates each config value with its source location.

Example Output Formats

Option 1: Comment-style annotation (works with -flat)

process.memory = { 6.GB * task.attempt }  // /Users/ewels/GitHub/nf-core/rnaseq/conf/test.config
process.time = { 4.h * task.attempt }  // /Users/ewels/GitHub/nf-core/rnaseq/conf/test.config
process.errorStrategy = { task.exitStatus in ((130..145) + 104 + 175) ? 'retry' : 'finish' }  // /Users/ewels/GitHub/nf-core/rnaseq/nextflow.config
process.maxRetries = 1 // /Users/ewels/GitHub/nf-core/rnaseq/nextflow.config
process.'withLabel:process_low'.memory = { 12.GB * task.attempt } // /data/working/launchdir/nextflow.config
process.'withLabel:process_low'.time = { 4.h * task.attempt } // [command line]

Option 2: Git-blame style left sidebar

/Users/ewels/GitHub/nf-core/rnaseq/conf/test.config | process.memory = { 6.GB * task.attempt }
/Users/ewels/GitHub/nf-core/rnaseq/conf/test.config | process.time = { 4.h * task.attempt }
/Users/ewels/GitHub/nf-core/rnaseq/nextflow.config  | process.errorStrategy = { task.exitStatus in ((130..145) + 104 + 175) ? 'retry' : 'finish' }

Implementation Notes

The ConfigBuilder.parse0() method already tracks parsed config files via parsedConfigFiles. The implementation would need to:

  1. Track the source file/origin for each config value as configs are parsed and merged
  2. Preserve this attribution through the merge process (later sources override earlier ones)
  3. Add a new CLI flag to the config command to enable blame output
  4. Format and display the attribution in the output
Additional Considerations
  • Trace logging: Consider also adding trace-level logging in ConfigBuilder to show configs as they're loaded during actual pipeline runs (useful for debugging Platform-injected configs)
  • Platform integration: When configs come from Seqera Platform, ideally the attribution would distinguish between different Platform config sources (Compute Environment, Workspace, etc.). This may require Platform to pass multiple config files or include blame metadata.

Related Discussion

Slack thread: https://seqera.slack.com/archives/CUTDS6JE9/p1772703970085559?thread_ts=1772617083.053159&cid=CUTDS6JE9

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with ConfigBuilder.parse0() and its parsedConfigFiles tracking, then inspect how the config command formats merged values. Define how attribution survives config merges and how a new --blame flag selects the output format. Done means config values show the winning source or origin, including command-line values, in the selected output.

Written by the indexing model from the issue text.

Assessment

Tech stack
groovy
Domain
cli, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.