MetOffice / MetOffice/XBTs_classification

optmisation of training, inference and evaluation tasks

Open
#44 0 comments 0 reactions 0 assignees View on GitHub
performance
Dominant language
Jupyter Notebook
Stars
4
Forks
2
PR merge metrics
No merged PRs in 30d

Description

A lot of the XBT code feels quite slow. I don't have a benchmark so I don't know whther it actually is slow, but the parts that are slow seem simpler than the parts that are fast e.g. algorithm training is quicker than running iMeta, so some of the slow parts can conceivably be expedited. Things that should be looked at to improve performance:
* loading data
* running iMeta algorithm
* calculating splits
* outputting data

In terms of data, it might be that another data format would be quicker e.g. [parquet](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_parquet.html) .

Some things are just inherently slow, but wiaiting times could be reduced by parallel processing. Area which could be run in parallel include:
* hyperparameter tuning - parallelism using joblib/dask
* some [advice on using dask with joblib](https://ml.dask.org/joblib.html) for parallelisation
* netcdf to csv conversion - this seems like a good task for dask based parallelism
* metric calculation - splitting into years for dask calculations

A starting point is to measure performance on different platforms for each section of the algorithm. Then compare performance for single for parallel execution.

Contributor guide

No contributing guide indexed for this repository

Research direction

No files, tests, or entry points are named. Start by measuring loading data, iMeta, split calculation, and output performance on different platforms, then compare single-process and parallel execution. Done means the slow sections and any measurable gains from alternatives such as Parquet or parallel processing are documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook, pandas
Domain
data, machine-learning, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.