snakemake / snakemake/snakemake
Job Grouping for local execution
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.9k
- Forks
- 656
- Avg merge
- 4d 12h
- Merged PRs (30d)
- 9
Description
My pipeline processes an input file and perform analysis for each sample in the file.
The overall outline is this:
checkpoint samples - reads input creates {sample}.txt for every sample
rule prepare_data - writes temporary data file for sample temp("{sample}_data.txt")
rule run_analysis - run analysis using the temporary data file, generates {sample}_output
rule aggregate - aggregates all sample results {sample}_output and create "fininished.txt"
The workflow can be locally executed; however, it is very costly to scale to higher number of samples because snakemake performs the rule prepare_data for all samples first before performing rule run_analysis. I end up having hundreds of temporary data files. I want to group prepare_data and run_analysis together so that the temporary data file can be used and deleted as we analyze each sample, but this grouping functionality is disabled for local execution. Is there a way to enable job grouping for local execution?
I have tried assign rule run_analysis priority 2 and rule prepare_data priority 1 and used --set-threads prepare_data=1 but the order of jobs do not change.
I tried assigning group "s" to both run_analysis and prepare_data but it has no effect as expected.
I cannot use the pipe("{sample}_data.txt") because the prepare_data step is a python script and not a shell command.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Begin by investigating how local execution handles grouped jobs, using the prepare_data and run_analysis rules described in the issue as the reproduction case. Done means those rules can be grouped for local execution so each sample's temporary data is removed after analysis without requiring a shell pipe.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100