khanlab / khanlab/snakebids

Automatically create pybidsdb

Open
#256 0 comments 0 reactions 0 assignees View on GitHub
enhancement plugin
Dominant language
Python
Stars
23
Forks
16
PR merge metrics
No merged PRs in 30d

Description

In recent versions of `snakemake`, the app will start subinstances of snakemake under many different circumstances, including cluster submission, use of the `run` directive, use of `shadow`, and probably others. Every time these sub-instances are started, `generate_inputs` will be called and, if no database exists, the dataset will be indexed. For large datasets, this is a huge problem, as indexing can take an immense amount of time (even if just a few minutes, it's still multiplied over the hundreds of times `generate_inputs` could potentially be called). Thus, for large datasets and complex workflow, a pybids database is essentially mandatory.

We could just expect users to make their own pybids database, but for downstream users, this is inconvenient. I thus propose a mechanism for automatically making a database in the case no database is provided.

I would probably conceive of this as a snakebids "extra" (see #255), so it's something app developers would need to opt into. Basically, if no database is provided, then every time `run.py` is run, it will use pybids to index the dataset and put the database into the `.snakebids` folder (which we'd need to create) in the output dir. This database will be passed on to snakemake and `generate_inputs()`.

Importantly, every time `run.py` is called, the database will be regenerated, regardless of whether it exists. This re-creates the expectation that the dataset will be re-indexed every time the app is run.

Thus, if users want a persistent database across runs, they'll still need to make their own, but within a single run, we can take advantage of the speed boosts of a database.

One concern is *how* exactly to make the db. On NAS, this can be *extremely* slow. On AllianceCan, the `/tmp` dir is mounted in memory, and I would presume in general that `/tmp` dirs are on more readily accessible storage locations. So it might be enough to always make the database on `/tmp` then copy it to `.snakebids/`.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the run.py entry point and the existing extra mechanism proposed in #255, then trace how generate_inputs receives a database and how Snakemake sub-instances are launched. Define the opt-in behavior, .snakebids output location, regeneration rules, and handling of temporary storage before implementation; done means one database is created per run and passed through consistently.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
databases, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.