MeteoSwiss / MeteoSwiss/mlpp-features
Stable interface to mlpp-workflows
- Dominant language
- Python
- Stars
- 6
- Forks
- 0
- Avg merge
- 21m
- Merged PRs (30d)
- 2
Description
Every time we change the way pipelines are called (e.g. by changing function arguments) we have to adapt the code in mlpp-workflows accordingly. It would be better instead if we had a stable interface between the two libraries.
It could be in the form of an output `xr.Dataset` object.
This could be done by simply moving the following function ([defined](https://github.com/MeteoSwiss/mlpp-workflows/blob/e1c3cd72be30067d1c556742b0613e4b61090709/scripts/process_source.py#L168-L187) in mlpp-workflows) to this library.
```python
def extract_features(
data: Dict[str, xr.Dataset],
feature_list: List[str],
points: Tuple[List],
reftimes: List[datetime],
leadtimes: List[int],
) -> xr.Dataset:
"""Extract features from a given source."""
ds = xr.Dataset()
for feature in feature_list:
LOGGER.info(f"FEATURE: {feature}")
try:
output = getattr(globals()["mlpp_features"], feature)(
data, points, reftimes, leadtimes, ds=ds
)
except:
LOGGER.exception(f"{feature} pipeline failed!")
ds[feature] = output.chunk("auto").persist()
LOGGER.info(ds)
return ds
```
It will also be easier to document how the two libraries interact since it will be just one object.
@dnerini thoughts?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with extract_features in mlpp-workflows/scripts/process_source.py and trace how mlpp-features and mlpp-workflows exchange pipeline inputs and outputs. Define and document the stable xr.Dataset-based boundary, then verify that existing cross-library calls work without adapting to function-argument changes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend-api-design, machine-learning
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100