daisybio / daisybio/drevalpy

In-memory processing and CLI alignment

Open
#454 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
44
Forks
12
Avg merge
7d 19h
Merged PRs (30d)
10

Description

This only targets the python API:

Currently, most processing steps operate on-disk, meaning one provides an input file/directory and an output directory, in the beginning the content of the input is read, then the processing is done, and then the results are written.

In my opinion, this is not really a pythonic way of handling multi-step processes. I would rather have something like:

1. `load_dataset(path)` → `Dataset` class (this step reads from disk to memory)
2. `split(d: Dataset)` → Mutated `Dataset` class or new class
3. `construct_model(model_name)` → `Model` class
4. `train(model, d: Dataset)` → Trained `Model` class
5. `predict(model, d: Dataset)` → Predictions
6. Optional: `save_predictions(p: Predictions)` → Store predictions to disk
7. `eval(p: Predictions)` → `Evaluations` class
8. `plot(e: Evaluations, p: Predictions)` → Plots

The function names are only schematics, real function names will be different. But I think they are sufficient to understand the concept.

---

For the CLI, keeping intermediate results in memory is of course not possible. I imagine the CLI to be a collection of light wrappers around high-level functions as outlined above - which is kind of already the case in the current codebase, but the difference would be that all CLI endpoints would follow the same structure:
1. Typer for argparsing
2. Loading input data from disk (each class should come with its built-in from-disk loading)
3. Calling the equivalent python function, which is imported from other submodules
4. Storing results to disk (again, single method call on the result class)

So in summary, there would be a strict 1:1 matching between top-level python functions and CLI endpoints. This should ensure that there is no real logic in the `cli` submodule in the package. Currently there is quite a lot of logic there, and it's quite opaque to me, why this is needed.

From my POV, there should be a set of top-level python functions, for each of them an equivalent CLI enpoint should exist that adds no additional logic, and also a nextflow module that wraps the CLI endpoint. This way, the mental model is always the exact same, independent of the interface

Contributor guide

Open the contributing guide

Research direction

Start by inventorying the current Python API, cli submodule, Typer endpoints, and Nextflow wrappers to identify existing processing logic and their relationships. Define the high-level functions and result classes needed for the proposed in-memory flow, then verify that each has a matching thin CLI endpoint and wrapper with disk I/O confined to loading and saving.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, cli, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.