google-deepmind / google-deepmind/ithaca

Understanding Ithaca's Evaluation/Checkpointing System

Open
#5 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
579
Forks
65
PR merge metrics
No merged PRs in 30d

Description

Hello all,

I'm seeking clarification on how Ithaca's evaluation/checkpointing system works.

From my understanding, the [evaluate function](https://github.com/deepmind/ithaca/blob/a3cb18d37da9fdc8f9a50a32b4c68c0b8b084caa/train/experiment.py#L377) should calculate the evaluation metrics and store the checkpoint's pickle file on disk. However, I'm uncertain about when this function is called.

Currently, when I execute the code, I notice that it just generate a log file containing the training loss and the accuracy. However, it doesn't include information about the validation loss, nor a checkpoint is produced.

Also when I try to run:
`python3 experiment.py --config=config.py --jaxline_mode=eval --logtostderr`
it says:
`Checkpoint None invalid or already evaluated, waiting.`

Thank you for your time and assistance.

Best regards,
Alessandro

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.