AlexsLemonade / AlexsLemonade/sc-data-integration
Make setup_envs.sh work without input files
- Dominant language
- HTML
- Stars
- 0
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
> While starting to test this, I ran into something funky that I could be a bug in our setup but may just be something to be aware of that requires documentation.
Before going to test, I ended up deleting all files in `results/` since I wanted to start everything from scratch; things had previously gotten out of whack with canceled runs, so I decided to just clean out. As a consequence, the results generated from scripts that get run before the pipeline, namely the `scpca-downstream-analyses` results which serve as Snakefile input, did not exist. As a consequence, I was unable to run `setup_envs.sh`, which makes some sense -
```
» bash setup_envs.sh
On Apple Silicon: Building R environment and required R packages with CONDA_SUBDIR=osx-64 for compatibility
Detecting duplicate sample names ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
Building DAG of jobs...
Building conda environments
Detecting duplicate sample names ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
Building DAG of jobs...
WorkflowError:
MissingInputException: Missing input files for rule merge_sces:
output: results/human_cell_atlas/merged_sce
affected files:
results/human_cell_atlas/scpca-downstream-analyses
MissingInputException: Missing input files for rule convert_anndata_sce:
output: results/human_cell_atlas/integrated_sce/HumanBrainSubstantiaNigra_integrated_fastmnn_sce.rds
wildcards: basedir=results/human_cell_atlas, project=HumanBrainSubstantiaNigra, method=fastmnn
affected files:
results/human_cell_atlas/integrated_anndata/HumanBrainSubstantiaNigra_integrated_fastmnn.h5
```
> I wonder if we just need to add something into the README that you can't setup environments until downstream has been run? Or is something else happening here?
> Note that now I'm re-running `scripts/01-run-downstream-analyses.sh` now to be able to proceed to test this.
_Originally posted by @sjspielman in https://github.com/AlexsLemonade/sc-data-integration/issues/162#issuecomment-1259460463_
There are probably a few options for how to do this, though I have not tested their implementation. One is to create more dummy rules for other environments, which could be done with a little loop like the one described here: https://github.com/snakemake/snakemake/issues/352#issuecomment-629887403
Then we could specify that rule in the second part of the `setup_envs.sh` script rather than `--create-envs-only`.
Other methods welcome too!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with setup_envs.sh and the Snakefile, then review scripts/01-run-downstream-analyses.sh and the README to understand which inputs environment setup currently requires. Reproduce the failure with the downstream results absent and compare possible Snakemake rule approaches. Done means setup_envs.sh completes without requiring those generated input files.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- shell
- Domain
- build-system
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100