tensorflow / tensorflow/models
Save Check point while training checkpoint_every_n
@pkulzc is already working on this.
Since Jan 20, 2021.
- Dominant language
- Python
- Stars
- 77.7k
- Forks
- 44.8k
- PR merge metrics
- No merged PRs in 30d
Description
Prerequisites
Please answer the following questions for yourself before submitting an issue.
- [ True] I am using the latest TensorFlow Model Garden release and TensorFlow 2.
- [ research directory] I am reporting the issue to the correct repository. (Model Garden official or research directory)
- [ Yes] I checked to make sure that this issue has not already been filed.
1. The entire URL of the file you are using
https://github.com/tensorflow/models/tree/master/research/object_detection/model_lib_v2.py
2. Describe the bug
A clear and concise description of what the bug is.
there a parameter to save checkpoint through learning checkpoint_every_n
suppose this value was checkpoint_every_n = 1000 and i make the steps number to be 800 or 1200
actually if i set 800 there in no checkpoint will saved and the exporter export empty model will not able to predict anything
the other case last checkpoint will be only for 1000 and the other 200 will not saved and if i planned to resume a model with higher steps again it will start from 1000 not 1200
3. Steps to reproduce
mentioned above
Steps to reproduce the behavior.
4. Expected behavior
A clear and concise description of what you expected to happen.
5. Additional context
Include any logs that would be helpful to diagnose the problem.
if ((int(global_step.value()) - checkpointed_step) >=
checkpoint_every_n or global_step.value() == train_steps):
if(global_step.value() == train_steps) :
warning.warn("Last Step")
else:
warning.warn("Regular Check Point")
manager.save()
checkpointed_step = int(global_step.value())
i just recommended if we can save in case i reach final step or checkpoint_every_n
6. System information
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
- Mobile device name if the issue happens on a mobile device:
- TensorFlow installed from (source or binary):
- TensorFlow version (use command below):
- Python version:
- Bazel version (if compiling from source):
- GCC/Compiler version (if compiling from source):
- CUDA/cuDNN version:
- GPU model and memory:
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.