DiamondLightSource / DiamondLightSource/httomo

Sort out inconsistencies and oddities with the loader's YAML template and how a loader is referenced in a pipeline file

Open
#501 0 comments 1 reaction 0 assignees View on GitHub
loader refactor
Dominant language
Python
Stars
10
Forks
5
Avg merge
5d 17h
Merged PRs (30d)
1

Description

## Current situation with the loader and its YAML template

The history of the loader YAML template is a bit weird, so it may be worth summarising here:
- we had the `standard_tomo()` _function_ in the `httomo.data.hdf.loaders` module, and the YAML template for the loader was generated from that function
- when the block loader was introduced, namely, the `StandardTomoLoader` _class_ in the `httomo.loaders.standard_tomo_loader` module, we didn't know how to tackle generating a YAML template from the class, so we pushed the problem to the side and simply:
- kept `httomo.data.hdf.loaders` in the YAML template and pipeline
- in the refactoring of httomo where the block loader was introduced, we basically treat the very first method in the pipeline as a special case, and thus we do not adhere to the `module_path` and `method` config for the loader in the YAML pipeline

The situation we have now is that the YAML config for a loader is inconsistent with where the loader code is actually located (ie, it's in `httomo.loaders.standard_tomo_loader` rather than `httomo.data.hdf.loaders`, despite the config implying the latter).

## Stuff to address (they're somewhat connected, rather than being totally independent problems)

### Loader YAML template generation

Due to the block loader being a class and not a function, enabling the YAML template generator script to work with loaders which are classes and not functions will probably require some tweaks to it. For example, maybe by checking if `httomo.loaders` is part of the module path, loader config in the pipeline can be caught and then treated differently to methods (which are just functions)?

### Referencing a loader in a pipeline file

The old loader (ie, the loader which didn't load individual blocks) in `httomo.data.hdf.loaders` is not used anymore, yet pipelines all reference `httomo.data.hdf.loaders`. To be consistent with how methods in a YAML pipeline file are referenced via their:
- module path
- method function name

it would make sense to have the way to reference an httomo loader in a pipeline to be something like:
```yaml
- method: StandardTomoLoader
module_path: httomo.loaders.standard_tomo_loader
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.