MetOffice / MetOffice/XBTs_classification
Add metadata file for categories
@stevehadd is already working on this.
Since May 12, 2020.
- Dominant language
- Jupyter Notebook
- Stars
- 4
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
Currently we have to load the whole dataset to be sure we have captured all the categories for caztegorical features in the dataset, because if we split the dataset by year, we are not guarentted to have all categories in all years. If we set up encoding based on the contents of a particular year, this can be problematic when comparing results across years. We want the encodings to be the same across the dataset. Currently this involves loading the whole dataset. Instead we could create a metafile which consolidates this info with a list of classes for each categorical feature, and some information for scaling for other features.
This should make it easier to parallelise code as only this metadata file is needed at the start to set up processing pipelines in a library like dask.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.