MetOffice / MetOffice/ml_weather_tutorial

Improvements to hyperparameter tuning tutorial.

Open
#22 0 comments 0 reactions 1 assignee View on GitHub

@stevehadd is already working on this.

Since Nov 8, 2022.

Dominant language
Jupyter Notebook
Stars
4
Forks
2
PR merge metrics
No merged PRs in 30d

Description

From review by @HarveySouth
Could: extend hyperparameter tuning tutorial with a test set evaluation from the best model. It would be interesting to discuss, but my own idea of reporting a good estimate of model performance involves finding the best hyperparameters that maximize the desired metric on the validation set, and then applying exactly the same training process to the model, trained on the training and validation data aggregated, tested against the test set since step/epoch are hyperparameters the final model inference metric is reportable. It is of course possible to report just the whole training process (which should be done anyway) and leave interpretation to the user/reader, but not sure what a best practice in this case would look like.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.