In-memory processing and CLI alignment
- Dominant language
- Python
- Stars
- 44
- Forks
- 12
- Avg merge
- 7d 19h
- Merged PRs (30d)
- 10
Description
This only targets the python API:
Currently, most processing steps operate on-disk, meaning one provides an input file/directory and an output directory, in the beginning the content of the input is read, then the processing is done, and then the results are written.
In my opinion, this is not really a pythonic way of handling multi-step processes. I would rather have something like:
1. `load_dataset(path)` → `Dataset` class (this step reads from disk to memory)
2. `split(d: Dataset)` → Mutated `Dataset` class or new class
3. `construct_model(model_name)` → `Model` class
4. `train(model, d: Dataset)` → Trained `Model` class
5. `predict(model, d: Dataset)` → Predictions
6. Optional: `save_predictions(p: Predictions)` → Store predictions to disk
7. `eval(p: Predictions)` → `Evaluations` class
8. `plot(e: Evaluations, p: Predictions)` → Plots
The function names are only schematics, real function names will be different. But I think they are sufficient to understand the concept.
---
For the CLI, keeping intermediate results in memory is of course not possible. I imagine the CLI to be a collection of light wrappers around high-level functions as outlined above - which is kind of already the case in the current codebase, but the difference would be that all CLI endpoints would follow the same structure:
1. Typer for argparsing
2. Loading input data from disk (each class should come with its built-in from-disk loading)
3. Calling the equivalent python function, which is imported from other submodules
4. Storing results to disk (again, single method call on the result class)
So in summary, there would be a strict 1:1 matching between top-level python functions and CLI endpoints. This should ensure that there is no real logic in the `cli` submodule in the package. Currently there is quite a lot of logic there, and it's quite opaque to me, why this is needed.
From my POV, there should be a set of top-level python functions, for each of them an equivalent CLI enpoint should exist that adds no additional logic, and also a nextflow module that wraps the CLI endpoint. This way, the mental model is always the exact same, independent of the interface
Contributor guide
Research direction
Start by inventorying the current Python API, cli submodule, Typer endpoints, and Nextflow wrappers to identify existing processing logic and their relationships. Define the high-level functions and result classes needed for the proposed in-memory flow, then verify that each has a matching thin CLI endpoint and wrapper with disk I/O confined to loading and saving.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, cli, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100