MeteoSwiss / MeteoSwiss/mlpp-features

Stable interface to mlpp-workflows

Open
#14 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
6
Forks
0
Avg merge
21m
Merged PRs (30d)
2

Description

Every time we change the way pipelines are called (e.g. by changing function arguments) we have to adapt the code in mlpp-workflows accordingly. It would be better instead if we had a stable interface between the two libraries.

It could be in the form of an output `xr.Dataset` object.

This could be done by simply moving the following function ([defined](https://github.com/MeteoSwiss/mlpp-workflows/blob/e1c3cd72be30067d1c556742b0613e4b61090709/scripts/process_source.py#L168-L187) in mlpp-workflows) to this library.

```python
def extract_features(
data: Dict[str, xr.Dataset],
feature_list: List[str],
points: Tuple[List],
reftimes: List[datetime],
leadtimes: List[int],
) -> xr.Dataset:
"""Extract features from a given source."""
ds = xr.Dataset()
for feature in feature_list:
LOGGER.info(f"FEATURE: {feature}")
try:
output = getattr(globals()["mlpp_features"], feature)(
data, points, reftimes, leadtimes, ds=ds
)
except:
LOGGER.exception(f"{feature} pipeline failed!")
ds[feature] = output.chunk("auto").persist()
LOGGER.info(ds)
return ds
```

It will also be easier to document how the two libraries interact since it will be just one object.

@dnerini thoughts?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with extract_features in mlpp-workflows/scripts/process_source.py and trace how mlpp-features and mlpp-workflows exchange pipeline inputs and outputs. Define and document the stable xr.Dataset-based boundary, then verify that existing cross-library calls work without adapting to function-argument changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend-api-design, machine-learning
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.