google-deepmind / google-deepmind/ithaca
Understanding Ithaca's Evaluation/Checkpointing System
- Dominant language
- Jupyter Notebook
- Stars
- 579
- Forks
- 65
- PR merge metrics
- No merged PRs in 30d
Description
Hello all,
I'm seeking clarification on how Ithaca's evaluation/checkpointing system works.
From my understanding, the [evaluate function](https://github.com/deepmind/ithaca/blob/a3cb18d37da9fdc8f9a50a32b4c68c0b8b084caa/train/experiment.py#L377) should calculate the evaluation metrics and store the checkpoint's pickle file on disk. However, I'm uncertain about when this function is called.
Currently, when I execute the code, I notice that it just generate a log file containing the training loss and the accuracy. However, it doesn't include information about the validation loss, nor a checkpoint is produced.
Also when I try to run:
`python3 experiment.py --config=config.py --jaxline_mode=eval --logtostderr`
it says:
`Checkpoint None invalid or already evaluated, waiting.`
Thank you for your time and assistance.
Best regards,
Alessandro
Contributor guide
Assessment
This issue has not been assessed yet.