MetOffice / MetOffice/XBTs_classification
optmisation of training, inference and evaluation tasks
- Dominant language
- Jupyter Notebook
- Stars
- 4
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
A lot of the XBT code feels quite slow. I don't have a benchmark so I don't know whther it actually is slow, but the parts that are slow seem simpler than the parts that are fast e.g. algorithm training is quicker than running iMeta, so some of the slow parts can conceivably be expedited. Things that should be looked at to improve performance:
* loading data
* running iMeta algorithm
* calculating splits
* outputting data
In terms of data, it might be that another data format would be quicker e.g. [parquet](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_parquet.html) .
Some things are just inherently slow, but wiaiting times could be reduced by parallel processing. Area which could be run in parallel include:
* hyperparameter tuning - parallelism using joblib/dask
* some [advice on using dask with joblib](https://ml.dask.org/joblib.html) for parallelisation
* netcdf to csv conversion - this seems like a good task for dask based parallelism
* metric calculation - splitting into years for dask calculations
A starting point is to measure performance on different platforms for each section of the algorithm. Then compare performance for single for parallel execution.
Contributor guide
No contributing guide indexed for this repository
Research direction
No files, tests, or entry points are named. Start by measuring loading data, iMeta, split calculation, and output performance on different platforms, then compare single-process and parallel execution. Done means the slow sections and any measurable gains from alternatives such as Parquet or parallel processing are documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter-notebook, pandas
- Domain
- data, machine-learning, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100