Provided checkpoint files not sufficient to restore agent?
- Dominant language
- Jupyter Notebook
- Stars
- 10.9k
- Forks
- 1.4k
- PR merge metrics
- No merged PRs in 30d
Description
For a given game and run (say Qbert/1/) the following files are included:
- tf_ckpt-199.data-00000-of-00001
- tf_ckpt-199.index
- tf_ckpt-199.meta
However, the function dopamine.common.checkpointer.get_latest_checkpoint_number looks for files with `sentinel_checkpoint_complete.*` to determine the largest checkpoint file to load.
```
def get_latest_checkpoint_number(base_directory):
"""Returns the version number of the latest completed checkpoint.
Args:
base_directory: str, directory in which to look for checkpoint files.
Returns:
int, the iteration number of the latest checkpoint, or -1 if none was found.
"""
glob = os.path.join(base_directory, 'sentinel_checkpoint_complete.*')
def extract_iteration(x):
return int(x[x.rfind('.') + 1:])
try:
checkpoint_files = tf.gfile.Glob(glob)
except tf.errors.NotFoundError:
return -1
try:
latest_iteration = max(extract_iteration(x) for x in checkpoint_files)
return latest_iteration
except ValueError:
return -1
```
The list `checkpoint_files` is empty.
As a result the check on dopamine.atart.run_experiment.py:204 fails and unbundle, which would read in the provided checkpoint files, never gets called on the agent.
Contributor guide
Assessment
This issue has not been assessed yet.