google / google/uncertainty-baselines
Time elapsed msg does not account for time before any preemptions
Open
- Dominant language
- Python
- Stars
- 1.6k
- Forks
- 224
- Avg merge
- 15h 36m
- Merged PRs (30d)
- 2
Description
> @dustinvtran
> @ywen666
>
> We should save the time elapsed number in checkpoints. This enables pre-empted jobs to add on top of the number instead of restarting from 0 but with a later checkpoint.
>
> Looking at https://www.tensorflow.org/api_docs/python/tf/train/Checkpoint, this requires storing `start_time` as `tf.Variable(start_time)`.
https://github.com/google/edward2/issues/260
Contributor guide
Assessment
This issue has not been assessed yet.