Find exported checkpoints when specified by step number.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39
- Forks
- 7
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 5
Description
Summary:
Find exported checkpoints when specified by step number.
Details:
When translating I can specify checkpoints best or last or by step number for the model. Sometimes there is a checkpoint which isn't found when the user specifies the step-number for that checkpoint.
Example:
For a particular model the best checkpoint was exported at 7000 steps. The last checkpoint was at 11000 steps and that checkpoint remains in the run folder. The config file specified that only one checkpoint is to be saved.
train:
keep_checkpoint_max: 1
save_checkpoints_steps: 1000
That results in the following folder structure:
MT/experiments//run/export/7000/ckpt.data-00000-of-00001
MT/experiments//run/export/7000/ckpt.index
MT/experiments//run/checkpoint
MT/experiments//run/ckpt-11000.data-00000-of-00001
MT/experiments//run/ckpt-11000.index
These commands work:
python -m silnlp.nmt.translate --checkpoint best ... (Finds checkpoint 7000)
python -m silnlp.nmt.translate --checkpoint last ... (Finds checkpoint 11000)
python -m silnlp.nmt.translate --checkpoint 11000 ... (Finds checkpoint 11000)
This command fails to find checkpoint 7000:
python -m silnlp.nmt.translate --checkpoint 7000
With this error:
Traceback (most recent call last):
File "/home/david/.cache/pypoetry/virtualenvs/silnlp-gt_VMn9E-py3.8/lib/python3.8/site-packages/tensorflow/python/training/py_checkpoint_reader.py", line 95, in NewCheckpointReader
return CheckpointReader(compat.as_bytes(filepattern))
RuntimeError: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for /home/david/gutenberg/MT/experiments/BT-Spanish/nhx-es-2021_08_18_names_4/run/ckpt-7000
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/david/.cache/pypoetry/virtualenvs/silnlp-gt_VMn9E-py3.8/lib/python3.8/site-packages/tensorflow/python/training/tracking/util.py", line 2260, in restore
status = self.read(save_path, options=options)
File "/home/david/.cache/pypoetry/virtualenvs/silnlp-gt_VMn9E-py3.8/lib/python3.8/site-packages/tensorflow/python/training/tracking/util.py", line 2148, in read
return self._saver.restore(save_path=save_path, options=options)
File "/home/david/.cache/pypoetry/virtualenvs/silnlp-gt_VMn9E-py3.8/lib/python3.8/site-packages/tensorflow/python/training/tracking/util.py", line 1292, in restore
reader = py_checkpoint_reader.NewCheckpointReader(save_path)
File "/home/david/.cache/pypoetry/virtualenvs/silnlp-gt_VMn9E-py3.8/lib/python3.8/site-packages/tensorflow/python/training/py_checkpoint_reader.py", line 99, in NewCheckpointReader
error_translator(e)
File "/home/david/.cache/pypoetry/virtualenvs/silnlp-gt_VMn9E-py3.8/lib/python3.8/site-packages/tensorflow/python/training/py_checkpoint_reader.py", line 35, in error_translator
raise errors_impl.NotFoundError(None, None, error_message)
tensorflow.python.framework.errors_impl.NotFoundError: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for /home/david/gutenberg/MT/experiments/BT-Spanish/nhx-es-2021_08_18_names_4/run/ckpt-7000
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/david/silnlp/silnlp/nmt/translate.py", line 137, in <module>
main()
File "/home/david/silnlp/silnlp/nmt/translate.py", line 107, in main
translator.translate_book(src_project, book, output_path, trg_iso=trg_iso)
File "/home/david/silnlp/silnlp/common/translator.py", line 174, in translate_book
translations = list(self.translate((s.text.strip() for s in segments), src_iso=src_iso, trg_iso=trg_iso))
File "/home/david/silnlp/silnlp/nmt/translate.py", line 28, in translate
translations = self._runner.infer_list(features_list, checkpoint_path=str(self._checkpoint_path))
File "/home/david/silnlp/silnlp/nmt/runner.py", line 291, in infer_list
checkpoint.restore(checkpoint_path=checkpoint_path, weights_only=True)
File "/home/david/.cache/pypoetry/virtualenvs/silnlp-gt_VMn9E-py3.8/lib/python3.8/site-packages/opennmt/utils/checkpoint.py", line 147, in restore
load_status = checkpoint.restore(checkpoint_path)
File "/home/david/.cache/pypoetry/virtualenvs/silnlp-gt_VMn9E-py3.8/lib/python3.8/site-packages/tensorflow/python/training/tracking/util.py", line 2262, in restore
raise errors_impl.NotFoundError(
tensorflow.python.framework.errors_impl.NotFoundError: Could not find checkpoint or SavedModel at /home/david/gutenberg/MT/experiments/BT-Spanish/nhx-es-2021_08_18_names_4/run/ckpt-7000.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in silnlp/nmt/translate.py and follow checkpoint handling into silnlp/nmt/runner.py. Compare how numeric checkpoints are resolved with the best and last cases, using the reported export/7000 and run/ckpt-11000 paths. Done means --checkpoint 7000 finds the exported checkpoint while the existing numeric and named cases continue to work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, tensorflow
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100