Installing nf-core subworkflows causes error when subworkflow from other organization is present in modules.json
Open
@edmundmiller is already working on this.
Since Apr 3, 2025.
bug
- Dominant language
- Python
- Stars
- 322
- Forks
- 255
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 5
Description
Description of the bug
When working with subworkflows from another remote and another organization than nf-core/modules I run into issues.
This is my modules.json
{
"name": "ORGANIZATION/mypipeline",
"homePage": "https://github.com/<user>/mypipeline",
"repos": {
"git@bitbucket.org:mybitbucket/modules.git": {
"modules": {
"my_org": {
"deseq2/gene": {
"branch": "main",
"git_sha": "7c011fb5ef28e2454ade9b3b587d25c3b1eda03f",
"installed_by": [
"cross_org_subworkflow"
]
}
}
},
"subworkflows": {
"my_org": {
"cross_org_subworkflow": {
"branch": "main",
"git_sha": "7c011fb5ef28e2454ade9b3b587d25c3b1eda03f",
"installed_by": [
"pipeline-component-installer"
]
}
}
}
},
"https://github.com/nf-core/modules.git": {
"modules": {
"nf-core": {
"samtools/sort": {
"branch": "master",
"git_sha": "f1139432f654e4bd1ebd674861039ddb16150e9d",
"installed_by": [
"cross_org_subworkflow"
]
},
"samtools/index": {
"branch": "master",
"git_sha": "f1139432f654e4bd1ebd674861039ddb16150e9d",
"installed_by": [
"cross_org_subworkflow"
]
}
}
}
}
}
}
When I wish to install an nf-core subworkflow via the nf-core subworkflows install command I get an error (see the "Command used and terminal output" section).
If I remove the "subworkflows" key from "git@bitbucket.org:mybitbucket/modules.git", this error does not occur, however the "subworkflows" key from "git@bitbucket.org:mybitbucket/modules.git" seems to be absent/ ignored by the program when running the command.
Also, if I have not set the key entry for "git@bitbucket.org:mybitbucket/modules.git" in modules.json at all, before running the command, I will get no breakage when running the command and the modules.json is build correctly for nf-core remote, but not entirely for "git@bitbucket.org:mybitbucket/modules.git". When I run without pre-building the modules.json keys I will have to supply my remote "git@bitbucket.org:mybitbucket/modules.git" for where the locally installed modules/subworkflows of my_org are located and the program will automatically identify modules correctly and write this to modules.json, but not subworkflows, which will just be absent from the modules.json for "git@bitbucket.org:mybitbucket/modules.git".
I have not been able to identify the issue causing this error.
Command used and terminal output
Command run when subworkflows entry for mybitbucket remote is present in modules.json
$ nf-core --verbose --log-file test_log.txt subworkflows install fastq_align_bwa
,--./,-.
___ __ __ __ ___ /,-._.--~\
|\ | |__ __ / ` / \ |__) |__ } {
| \| | \__, \__/ | \ |___ \`-._,-`-,
`._,._,'
nf-core/tools version 3.2.0 - https://nf-co.re
DEBUG Failed checking if running in CYGWIN due to: FileNotFoundError(2, 'No such file or directory') util.py:472
Pulling from 'nf-core/modules' (https://github.com/nf-core/modules.git) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Waiting for responseDEBUG Popen(['git', 'version'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'fetch', '-v', '--progress', '--', 'origin'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=True) cmd.py:1253
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'merge', 'origin/master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Got '/home/usr_name/.config/nfcore/nf-core/modules' as path utils.py:280
DEBUG Found a config cache, loading: /home/usr_name/.nextflow/nf-core/wf-config-cache-9f4bce35ba6f9f759cbf54fa7.json utils.py:314
DEBUG Using config file: /home/usr_name/.config/nfcore/nf-core/modules/.nf-core.yml utils.py:1385
DEBUG Got '.' as path utils.py:280
DEBUG Found a config cache, loading: /home/usr_name/.nextflow/nf-core/wf-config-cache-380ad3d9d31216b384383e94b.json utils.py:314
DEBUG Using config file: .nf-core.yml utils.py:1385
INFO Recomputing commit SHAs for modules which were missing from 'modules.json': 'samtools/flagstat', 'samtools/stats', 'samtools/idxstats', 'bwa/mem' modules_json.py:1200
Pulling from 'my_bitbucket/modules' (git@bitbucket.org:my_bitbucket/modules.git) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Waiting for responseDEBUG Popen(['git', 'version'], cwd=/home/usr_name/.config/nfcore/my_bitbucket/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'fetch', '-v', '--progress', '--', 'origin'], cwd=/home/usr_name/.config/nfcore/my_bitbucket/modules, stdin=None, shell=False, universal_newlines=True) cmd.py:1253
DEBUG Popen(['git', 'checkout', 'main'], cwd=/home/usr_name/.config/nfcore/my_bitbucket/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'merge', 'origin/main'], cwd=/home/usr_name/.config/nfcore/my_bitbucket/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Got '/home/usr_name/.config/nfcore/my_bitbucket/modules' as path utils.py:280
DEBUG Found a config cache, loading: /home/usr_name/.nextflow/nf-core/wf-config-cache-383663e076355045a1d194025.json utils.py:314
DEBUG Using config file: /home/usr_name/.config/nfcore/my_bitbucket/modules/.nf-core.yml utils.py:1385
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'merge', 'origin/master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Got '/home/usr_name/.config/nfcore/nf-core/modules' as path utils.py:280
DEBUG Found a config cache, loading: /home/usr_name/.nextflow/nf-core/wf-config-cache-9f4bce35ba6f9f759cbf54fa7.json utils.py:314
DEBUG Using config file: /home/usr_name/.config/nfcore/nf-core/modules/.nf-core.yml utils.py:1385
DEBUG Popen(['git', 'checkout', 'main'], cwd=/home/usr_name/.config/nfcore/my_bitbucket/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'merge', 'origin/main'], cwd=/home/usr_name/.config/nfcore/my_bitbucket/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Got '/home/usr_name/.config/nfcore/my_bitbucket/modules' as path utils.py:280
DEBUG Found a config cache, loading: /home/usr_name/.nextflow/nf-core/wf-config-cache-383663e076355045a1d194025.json utils.py:314
DEBUG Using config file: /home/usr_name/.config/nfcore/my_bitbucket/modules/.nf-core.yml utils.py:1385
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'merge', 'origin/master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Got '/home/usr_name/.config/nfcore/nf-core/modules' as path utils.py:280
DEBUG Found a config cache, loading: /home/usr_name/.nextflow/nf-core/wf-config-cache-9f4bce35ba6f9f759cbf54fa7.json utils.py:314
DEBUG Using config file: /home/usr_name/.config/nfcore/nf-core/modules/.nf-core.yml utils.py:1385
DEBUG Popen(['git', 'checkout', 'main'], cwd=/home/usr_name/.config/nfcore/my_bitbucket/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'merge', 'origin/main'], cwd=/home/usr_name/.config/nfcore/my_bitbucket/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Got '/home/usr_name/.config/nfcore/my_bitbucket/modules' as path utils.py:280
DEBUG Found a config cache, loading: /home/usr_name/.nextflow/nf-core/wf-config-cache-383663e076355045a1d194025.json utils.py:314
DEBUG Using config file: /home/usr_name/.config/nfcore/my_bitbucket/modules/.nf-core.yml utils.py:1385
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'merge', 'origin/master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Got '/home/usr_name/.config/nfcore/nf-core/modules' as path utils.py:280
DEBUG Found a config cache, loading: /home/usr_name/.nextflow/nf-core/wf-config-cache-9f4bce35ba6f9f759cbf54fa7.json utils.py:314
DEBUG Using config file: /home/usr_name/.config/nfcore/nf-core/modules/.nf-core.yml utils.py:1385
DEBUG Popen(['git', 'checkout', 'main'], cwd=/home/usr_name/.config/nfcore/my_bitbucket/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'merge', 'origin/main'], cwd=/home/usr_name/.config/nfcore/my_bitbucket/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Got '/home/usr_name/.config/nfcore/my_bitbucket/modules' as path utils.py:280
DEBUG Found a config cache, loading: /home/usr_name/.nextflow/nf-core/wf-config-cache-383663e076355045a1d194025.json utils.py:314
DEBUG Using config file: /home/usr_name/.config/nfcore/my_bitbucket/modules/.nf-core.yml utils.py:1385
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'merge', 'origin/master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Got '/home/usr_name/.config/nfcore/nf-core/modules' as path utils.py:280
DEBUG Found a config cache, loading: /home/usr_name/.nextflow/nf-core/wf-config-cache-9f4bce35ba6f9f759cbf54fa7.json utils.py:314
DEBUG Using config file: /home/usr_name/.config/nfcore/nf-core/modules/.nf-core.yml utils.py:1385
DEBUG Popen(['git', 'checkout', 'main'], cwd=/home/usr_name/.config/nfcore/my_bitbucket/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'merge', 'origin/main'], cwd=/home/usr_name/.config/nfcore/my_bitbucket/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Got '/home/usr_name/.config/nfcore/my_bitbucket/modules' as path utils.py:280
DEBUG Found a config cache, loading: /home/usr_name/.nextflow/nf-core/wf-config-cache-383663e076355045a1d194025.json utils.py:314
DEBUG Using config file: /home/usr_name/.config/nfcore/my_bitbucket/modules/.nf-core.yml utils.py:1385
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'merge', 'origin/master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Got '/home/usr_name/.config/nfcore/nf-core/modules' as path utils.py:280
DEBUG Found a config cache, loading: /home/usr_name/.nextflow/nf-core/wf-config-cache-9f4bce35ba6f9f759cbf54fa7.json utils.py:314
DEBUG Using config file: /home/usr_name/.config/nfcore/nf-core/modules/.nf-core.yml utils.py:1385
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'merge', 'origin/master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Got '/home/usr_name/.config/nfcore/nf-core/modules' as path utils.py:280
DEBUG Found a config cache, loading: /home/usr_name/.nextflow/nf-core/wf-config-cache-9f4bce35ba6f9f759cbf54fa7.json utils.py:314
DEBUG Using config file: /home/usr_name/.config/nfcore/nf-core/modules/.nf-core.yml utils.py:1385
DEBUG Popen(['git', 'ls-remote', 'https://github.com/nf-core/modules.git'], cwd=/home/usr_name/internal_nf/nf-template, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'cat-file', '--batch-check'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=<valid stream>, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'rev-list', '--max-count=1000', '7658b5767a5d62656d23ccd6779dcd64a46a5773', '--', 'modules/nf-core/bwa/mem'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, cmd.py:1253
stdin=None, shell=False, universal_newlines=False)
DEBUG Popen(['git', 'rev-list', '--max-count=1000', '7658b5767a5d62656d23ccd6779dcd64a46a5773', '--', 'modules/bwa/mem'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, cmd.py:1253
shell=False, universal_newlines=False)
DEBUG Popen(['git', 'cat-file', '--batch'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=<valid stream>, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'checkout', 'a29f18660f5e3748d44d6f716241e70c942c065d'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'rev-list', '--max-count=1000', '7658b5767a5d62656d23ccd6779dcd64a46a5773', '--', 'modules/nf-core/samtools/flagstat'], cmd.py:1253
cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)
DEBUG Popen(['git', 'rev-list', '--max-count=1000', '7658b5767a5d62656d23ccd6779dcd64a46a5773', '--', 'modules/samtools/flagstat'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, cmd.py:1253
stdin=None, shell=False, universal_newlines=False)
DEBUG Popen(['git', 'checkout', '05954dab2ff481bcb999f24455da29a5828af08d'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'rev-list', '--max-count=1000', '7658b5767a5d62656d23ccd6779dcd64a46a5773', '--', 'modules/nf-core/samtools/idxstats'], cmd.py:1253
cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)
DEBUG Popen(['git', 'rev-list', '--max-count=1000', '7658b5767a5d62656d23ccd6779dcd64a46a5773', '--', 'modules/samtools/idxstats'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, cmd.py:1253
stdin=None, shell=False, universal_newlines=False)
DEBUG Popen(['git', 'checkout', '05954dab2ff481bcb999f24455da29a5828af08d'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'rev-list', '--max-count=1000', '7658b5767a5d62656d23ccd6779dcd64a46a5773', '--', 'modules/nf-core/samtools/stats'], cmd.py:1253
cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False)
DEBUG Popen(['git', 'rev-list', '--max-count=1000', '7658b5767a5d62656d23ccd6779dcd64a46a5773', '--', 'modules/samtools/stats'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, cmd.py:1253
stdin=None, shell=False, universal_newlines=False)
DEBUG Popen(['git', 'checkout', '05954dab2ff481bcb999f24455da29a5828af08d'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
DEBUG Popen(['git', 'checkout', 'master'], cwd=/home/usr_name/.config/nfcore/nf-core/modules, stdin=None, shell=False, universal_newlines=False) cmd.py:1253
{'git@bitbucket.org:my_bitbucket/modules.git': {'modules': {'my_org': {'deseq2/gene': {'branch': 'main', 'git_sha': '7c011fb5ef28e2454ade9b3b587d25c3b1eda03f', 'installed_by': ['cross_org_subworkflow']}}}, 'subworkflows': {'my_org': {'cross_org_subworkflow': {'branch': 'main', 'git_sha': '7c011fb5ef28e2454ade9b3b587d25c3b1eda03f', 'installed_by': ['pipeline-component-installer']}}}}, 'https://github.com/nf-core/modules.git': {'modules': {'nf-core': {'samtools/sort': {'branch': 'master', 'git_sha': 'f1139432f654e4bd1ebd674861039ddb16150e9d', 'installed_by': ['cross_org_subworkflow']}, 'samtools/index': {'branch': 'master', 'git_sha': 'f1139432f654e4bd1ebd674861039ddb16150e9d', 'installed_by': ['cross_org_subworkflow']}, 'bwa/mem': {'branch': 'master', 'git_sha': 'a29f18660f5e3748d44d6f716241e70c942c065d', 'installed_by': ['modules']}, 'samtools/flagstat': {'branch': 'master', 'git_sha': '05954dab2ff481bcb999f24455da29a5828af08d', 'installed_by': ['modules']}, 'samtools/idxstats': {'branch': 'master', 'git_sha': '05954dab2ff481bcb999f24455da29a5828af08d', 'installed_by': ['modules']}, 'samtools/stats': {'branch': 'master', 'git_sha': '05954dab2ff481bcb999f24455da29a5828af08d', 'installed_by': ['modules']}}}}}
git@bitbucket.org:my_bitbucket/modules.git modules my_org samtools/sort installed_by
ERROR 'samtools/sort'
System information
System information
Nextflow version: 24.10.5.5935
Hardware: HPC
Executor: Local
OS: Ubuntu server
Version of nf-core/tools: 3.2.0
Python version: 3.12.5
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.