Lack of determinism in results across identical runs
Nobody has claimed this yet.
- Dominant language
- Nextflow
- Stars
- 63
- Forks
- 39
- PR merge metrics
- No merged PRs in 30d
Description
Description of the bug
I’ve noticed a concerning lack of reproducibility when running the pipeline. Even when using the exact same input data and parameters across identical runs, the results change significantly. In some cases, I’ve seen a gene's FDR jump from 0.06 to 0.6 between two runs without a single change in the configuration.
I first tracked this down to the CRISPRCleanR module, where the output files differ every time despite keeping parameters like min_reads or min_targeted_genes constant. This inconsistency then propagates downstream, making the final MAGeCK rankings and statistics completely unreliable for a stable analysis.
It seems like there are stochastic processes in the R scripts or the MAGeCK steps that aren't being pinned down with a fixed seed. Is there any way to set a seed or any specific configuration to ensure the pipeline behaves deterministically? I’d like to know if this is a known issue or if there's a workaround to get consistent results across runs.
Command used and terminal output
Relevant files
No response
System information
Singularity
MAGECK RRA & MLE
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the pipeline with identical inputs and parameters, then inspect the CRISPRCleanR output and the MAGeCK RRA and MLE steps for the source of variation. Check the R scripts and MAGeCK configuration for stochastic behavior or seed handling. Done means repeated runs produce identical outputs and stable FDR and ranking results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- bioinformatics
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100