nf-core / nf-core/crisprseq

Lack of determinism in results across identical runs

Open
#269 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Nextflow
Stars
63
Forks
39
PR merge metrics
No merged PRs in 30d

Description

Description of the bug

I’ve noticed a concerning lack of reproducibility when running the pipeline. Even when using the exact same input data and parameters across identical runs, the results change significantly. In some cases, I’ve seen a gene's FDR jump from 0.06 to 0.6 between two runs without a single change in the configuration.

I first tracked this down to the CRISPRCleanR module, where the output files differ every time despite keeping parameters like min_reads or min_targeted_genes constant. This inconsistency then propagates downstream, making the final MAGeCK rankings and statistics completely unreliable for a stable analysis.

It seems like there are stochastic processes in the R scripts or the MAGeCK steps that aren't being pinned down with a fixed seed. Is there any way to set a seed or any specific configuration to ensure the pipeline behaves deterministically? I’d like to know if this is a known issue or if there's a workaround to get consistent results across runs.

Command used and terminal output

Relevant files

No response

System information

Singularity
MAGECK RRA & MLE

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the pipeline with identical inputs and parameters, then inspect the CRISPRCleanR output and the MAGeCK RRA and MLE steps for the source of variation. Check the R scripts and MAGeCK configuration for stochastic behavior or seed handling. Done means repeated runs produce identical outputs and stable FDR and ranking results.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
bioinformatics
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.