snakemake / snakemake/snakemake-executor-plugin-slurm

Installing via bioconda installs incompatible version of slurm-jobstep executor

Open
#461 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
55
Forks
57
Avg merge
40m
Merged PRs (30d)
1

Description

Software Versions

snakemake: 9.21.0
conda: miniforge3/25.3.1
snakemake-executor-plugin-slurm: 2.6.1
slurm-wlm: 22.05.8

Full version info after installing using both conda and pip contained in env_list_<pip/conda>.txt.

Describe the bug
When installing the snakemake-executor-plugin-slurm and snakemake-executor-plugin-slurm-jobstep directly using conda, snakemake-executor-plugin-slurm-jobstep is installed in with version 0.4.0.
This leads to all array jobs crashing, as they try to call --slurm-jobstep-array-execs which was not introduced in that version yet.
When explicitly asking for a later version of snakemake-executor-plugin-slurm-jobstep, conda claims that snakemake-executor-plugin-slurm 2.6.1 requires snakemake-executor-plugin-slurm-jobstep 0.4.0.

When using pip to install snakemake-executor-plugin-slurm and snakemake-executor-plugin-slurm-jobstep, snakemake-executor-plugin-slurm-jobstep is correctly installed with version 0.6.0.

Logs
From logs/slurm_logs/60196711_1.log:

__main__.py: error: unrecognized arguments: --slurm-jobstep-array-execs=...

Minimal example

sandbox.tar.gz

Create environments snakemake-slurm-debug-<conda/pip> using

conda create -f environment_<conda/pip>.yml

To run the error producing version run

conda activate snakemake-slurm-debug-conda
snakemake --profile ./slurm_profile.yml --config pkgmgr="conda"

which runs an array job consisting of two jobs executing test.py from the rule_run_test and an additional rule which prints the package information.

Additional context
I've added the slurm log files of both the conda installed version (logs/slurm_logs/60196711_1/2.log) and the pip installed version (logs/slurm_logs/60190718_1/2.log).
env_list_<conda/pip>.txt and env_info_<conda/pip>.txt contain the output of the system_information rule, env_list_<conda/pip>.txt specifically the conda list dump.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by creating the environment from environment_<conda/pip>.yml and reproduce the failure with the supplied snakemake command and slurm_profile.yml. Compare the conda and pip package lists with env_list_<conda/pip>.txt, then use logs/slurm_logs/60196711_1.log to verify that the conda-installed array job accepts the required argument and matches the working pip setup.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.