sillsdev / sillsdev/silnlp

Find exported checkpoints when specified by step number.

Open
#73 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement needs to be revisited pipeline 6: infer
Dominant language
Python
Stars
39
Forks
7
Avg merge
1d 9h
Merged PRs (30d)
5

Description

Summary:
Find exported checkpoints when specified by step number.

Details:
When translating I can specify checkpoints best or last or by step number for the model. Sometimes there is a checkpoint which isn't found when the user specifies the step-number for that checkpoint.

Example:
For a particular model the best checkpoint was exported at 7000 steps. The last checkpoint was at 11000 steps and that checkpoint remains in the run folder. The config file specified that only one checkpoint is to be saved.

train:
  keep_checkpoint_max: 1
  save_checkpoints_steps: 1000

That results in the following folder structure:
MT/experiments//run/export/7000/ckpt.data-00000-of-00001
MT/experiments//run/export/7000/ckpt.index
MT/experiments//run/checkpoint
MT/experiments//run/ckpt-11000.data-00000-of-00001
MT/experiments//run/ckpt-11000.index

These commands work:

python -m silnlp.nmt.translate --checkpoint best ...    (Finds checkpoint 7000)
python -m silnlp.nmt.translate --checkpoint last ...    (Finds checkpoint 11000)
python -m silnlp.nmt.translate --checkpoint 11000 ...   (Finds checkpoint 11000)

This command fails to find checkpoint 7000:
python -m silnlp.nmt.translate --checkpoint 7000

With this error:

Traceback (most recent call last):
  File "/home/david/.cache/pypoetry/virtualenvs/silnlp-gt_VMn9E-py3.8/lib/python3.8/site-packages/tensorflow/python/training/py_checkpoint_reader.py", line 95, in NewCheckpointReader
    return CheckpointReader(compat.as_bytes(filepattern))
RuntimeError: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for /home/david/gutenberg/MT/experiments/BT-Spanish/nhx-es-2021_08_18_names_4/run/ckpt-7000

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/david/.cache/pypoetry/virtualenvs/silnlp-gt_VMn9E-py3.8/lib/python3.8/site-packages/tensorflow/python/training/tracking/util.py", line 2260, in restore
    status = self.read(save_path, options=options)
  File "/home/david/.cache/pypoetry/virtualenvs/silnlp-gt_VMn9E-py3.8/lib/python3.8/site-packages/tensorflow/python/training/tracking/util.py", line 2148, in read
    return self._saver.restore(save_path=save_path, options=options)
  File "/home/david/.cache/pypoetry/virtualenvs/silnlp-gt_VMn9E-py3.8/lib/python3.8/site-packages/tensorflow/python/training/tracking/util.py", line 1292, in restore
    reader = py_checkpoint_reader.NewCheckpointReader(save_path)
  File "/home/david/.cache/pypoetry/virtualenvs/silnlp-gt_VMn9E-py3.8/lib/python3.8/site-packages/tensorflow/python/training/py_checkpoint_reader.py", line 99, in NewCheckpointReader
    error_translator(e)
  File "/home/david/.cache/pypoetry/virtualenvs/silnlp-gt_VMn9E-py3.8/lib/python3.8/site-packages/tensorflow/python/training/py_checkpoint_reader.py", line 35, in error_translator
    raise errors_impl.NotFoundError(None, None, error_message)
tensorflow.python.framework.errors_impl.NotFoundError: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for /home/david/gutenberg/MT/experiments/BT-Spanish/nhx-es-2021_08_18_names_4/run/ckpt-7000

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/david/silnlp/silnlp/nmt/translate.py", line 137, in <module>
    main()
  File "/home/david/silnlp/silnlp/nmt/translate.py", line 107, in main
    translator.translate_book(src_project, book, output_path, trg_iso=trg_iso)
  File "/home/david/silnlp/silnlp/common/translator.py", line 174, in translate_book
    translations = list(self.translate((s.text.strip() for s in segments), src_iso=src_iso, trg_iso=trg_iso))
  File "/home/david/silnlp/silnlp/nmt/translate.py", line 28, in translate
    translations = self._runner.infer_list(features_list, checkpoint_path=str(self._checkpoint_path))
  File "/home/david/silnlp/silnlp/nmt/runner.py", line 291, in infer_list
    checkpoint.restore(checkpoint_path=checkpoint_path, weights_only=True)
  File "/home/david/.cache/pypoetry/virtualenvs/silnlp-gt_VMn9E-py3.8/lib/python3.8/site-packages/opennmt/utils/checkpoint.py", line 147, in restore
    load_status = checkpoint.restore(checkpoint_path)
  File "/home/david/.cache/pypoetry/virtualenvs/silnlp-gt_VMn9E-py3.8/lib/python3.8/site-packages/tensorflow/python/training/tracking/util.py", line 2262, in restore
    raise errors_impl.NotFoundError(
tensorflow.python.framework.errors_impl.NotFoundError: Could not find checkpoint or SavedModel at /home/david/gutenberg/MT/experiments/BT-Spanish/nhx-es-2021_08_18_names_4/run/ckpt-7000.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in silnlp/nmt/translate.py and follow checkpoint handling into silnlp/nmt/runner.py. Compare how numeric checkpoints are resolved with the best and last cases, using the reported export/7000 and run/ckpt-11000 paths. Done means --checkpoint 7000 finds the exported checkpoint while the existing numeric and named cases continue to work.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, tensorflow
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.