Multi-run SDRF?
- Dominant language
- Nextflow
- Stars
- 19
- Forks
- 35
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 5
Description
### Description of feature
Could we from a single `nextflow run --input example.sdrf.tsv`, spawn different `quantms` instances?
For example:
I collected 3 different kinds of cell types from the same tissue. I prefer searching for these tissues using a biology-specific library. Therefore I would like to create 3 instances of `quantmsdiann`, one for each celltype. This entails, 3x scripts, 3x sdrf tables, 3x params.yml, etc.
What if from a single SDRF, we could have 2 additional columns: `session_id` where it describes to which pipeline run instance that row should be separated to; and `param_path` where one can designate which parameters should apply for that row.
**Example:**
| sample_id | celltype | session_id | param_path |
| ---------- | ---------- | ---------- | ---------- |
| sample_1 | Cancer_cells | session_1 | Cancer_params.yml |
| sample_2 | Cancer_cells | session_1 | Cancer_params.yml |
| sample_3 | T-cells | session_2 | T_cell_params.yml |
| sample_4 | T-cells | session_2 | T_cell_params.yml |
This would create 2 different instances of the pipeline.
Besides this biological vignette, which would be great to have, at least for our lab. I think this also has applications for technical experiments.
I could imagine running the same samples with two different sets of parameters, or with two different libraries.
At the moment I am using `job arrays` from SLURM, and while they work, it is clunkier, and requires plenty of file staging.
Having a single SDRF file could abstract entire experimental setups into a single table.
Thoughts?
Contributor guide
Research direction
No files, tests, or entry points are named in the issue. Start by locating the workflow's SDRF input handling and define how session_id and param_path would create separate runs; done means one SDRF can reproduce the requested grouped instances and parameter sets.
Written by the indexing model from the issue text.
Assessment
- Domain
- data-engineering, devops
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100