cytomining / cytomining/profiling-recipe
Move from linear execution strategy to modular block design
- Dominant language
- Python
- Stars
- 11
- Forks
- 34
- PR merge metrics
- No merged PRs in 30d
Description
Currently, the recipe (as defined in jump-cellpainting/profiling-recipe#14) is linear, with each step progressing sequentially. For example, the normalization step happens before the feature selection step.
A use case came up in the JUMP project that we will want to apply different normalization steps to the same input file, and then process the same feature selection step to both normalization output files.
This process is akin to a block design, in which each pipeline step is performed if and only if a "block" is added to the yaml config file.
A couple of implications of this enhancement:
* We will need to perform a substantial refactor to introduce this change
* We should explore adding the execution steps to a workflow language.
* We will need to add functionality to specify the input file in each block.
* Explore adding [dask](https://dask.org/) to the mix to enable task-graph parallelization
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the current linear recipe implementation and how the YAML configuration identifies and passes inputs between normalization and feature selection. Review the linked recipe discussion, then assess workflow-language and Dask options. Done means the design supports branching blocks, explicit input files, and applying the same downstream step to multiple outputs, with tests covering that flow.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100