cytomining / cytomining/profiling-recipe

Move from linear execution strategy to modular block design

Open
#11 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
11
Forks
34
PR merge metrics
No merged PRs in 30d

Description

Currently, the recipe (as defined in jump-cellpainting/profiling-recipe#14) is linear, with each step progressing sequentially. For example, the normalization step happens before the feature selection step.

A use case came up in the JUMP project that we will want to apply different normalization steps to the same input file, and then process the same feature selection step to both normalization output files.

This process is akin to a block design, in which each pipeline step is performed if and only if a "block" is added to the yaml config file.

A couple of implications of this enhancement:
* We will need to perform a substantial refactor to introduce this change
* We should explore adding the execution steps to a workflow language.
* We will need to add functionality to specify the input file in each block.
* Explore adding [dask](https://dask.org/) to the mix to enable task-graph parallelization

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the current linear recipe implementation and how the YAML configuration identifies and passes inputs between normalization and feature selection. Review the linked recipe discussion, then assess workflow-language and Dask options. Done means the design supports branching blocks, explicit input files, and applying the same downstream step to multiple outputs, with tests covering that flow.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.