AlexsLemonade / AlexsLemonade/sc-data-integration

Make setup_envs.sh work without input files

Open
#166 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
HTML
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

> While starting to test this, I ran into something funky that I could be a bug in our setup but may just be something to be aware of that requires documentation.
Before going to test, I ended up deleting all files in `results/` since I wanted to start everything from scratch; things had previously gotten out of whack with canceled runs, so I decided to just clean out. As a consequence, the results generated from scripts that get run before the pipeline, namely the `scpca-downstream-analyses` results which serve as Snakefile input, did not exist. As a consequence, I was unable to run `setup_envs.sh`, which makes some sense -

```
» bash setup_envs.sh
On Apple Silicon: Building R environment and required R packages with CONDA_SUBDIR=osx-64 for compatibility
Detecting duplicate sample names ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
Building DAG of jobs...
Building conda environments
Detecting duplicate sample names ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
Building DAG of jobs...
WorkflowError:
MissingInputException: Missing input files for rule merge_sces:
output: results/human_cell_atlas/merged_sce
affected files:
results/human_cell_atlas/scpca-downstream-analyses
MissingInputException: Missing input files for rule convert_anndata_sce:
output: results/human_cell_atlas/integrated_sce/HumanBrainSubstantiaNigra_integrated_fastmnn_sce.rds
wildcards: basedir=results/human_cell_atlas, project=HumanBrainSubstantiaNigra, method=fastmnn
affected files:
results/human_cell_atlas/integrated_anndata/HumanBrainSubstantiaNigra_integrated_fastmnn.h5
```

> I wonder if we just need to add something into the README that you can't setup environments until downstream has been run? Or is something else happening here?

> Note that now I'm re-running `scripts/01-run-downstream-analyses.sh` now to be able to proceed to test this.

_Originally posted by @sjspielman in https://github.com/AlexsLemonade/sc-data-integration/issues/162#issuecomment-1259460463_

There are probably a few options for how to do this, though I have not tested their implementation. One is to create more dummy rules for other environments, which could be done with a little loop like the one described here: https://github.com/snakemake/snakemake/issues/352#issuecomment-629887403

Then we could specify that rule in the second part of the `setup_envs.sh` script rather than `--create-envs-only`.

Other methods welcome too!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with setup_envs.sh and the Snakefile, then review scripts/01-run-downstream-analyses.sh and the README to understand which inputs environment setup currently requires. Reproduce the failure with the downstream results absent and compare possible Snakemake rule approaches. Done means setup_envs.sh completes without requiring those generated input files.

Written by the indexing model from the issue text.

Assessment

Tech stack
shell
Domain
build-system
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.