tensorflow / tensorflow/models

Save Check point while training checkpoint_every_n

Open
#9,650 4 comments 0 reactions 3 assignees View on GitHub

@pkulzc is already working on this.

Since Jan 20, 2021.

models:research:odapi type:bug
Dominant language
Python
Stars
77.7k
Forks
44.8k
PR merge metrics
No merged PRs in 30d

Description

Prerequisites

Please answer the following questions for yourself before submitting an issue.

  • [ True] I am using the latest TensorFlow Model Garden release and TensorFlow 2.
  • [ research directory] I am reporting the issue to the correct repository. (Model Garden official or research directory)
  • [ Yes] I checked to make sure that this issue has not already been filed.

1. The entire URL of the file you are using

https://github.com/tensorflow/models/tree/master/research/object_detection/model_lib_v2.py

2. Describe the bug

A clear and concise description of what the bug is.
there a parameter to save checkpoint through learning checkpoint_every_n
suppose this value was checkpoint_every_n = 1000 and i make the steps number to be 800 or 1200
actually if i set 800 there in no checkpoint will saved and the exporter export empty model will not able to predict anything
the other case last checkpoint will be only for 1000 and the other 200 will not saved and if i planned to resume a model with higher steps again it will start from 1000 not 1200

3. Steps to reproduce

mentioned above
Steps to reproduce the behavior.

4. Expected behavior

A clear and concise description of what you expected to happen.

5. Additional context

Include any logs that would be helpful to diagnose the problem.
if ((int(global_step.value()) - checkpointed_step) >= checkpoint_every_n or global_step.value() == train_steps): if(global_step.value() == train_steps) : warning.warn("Last Step") else: warning.warn("Regular Check Point") manager.save() checkpointed_step = int(global_step.value())
i just recommended if we can save in case i reach final step or checkpoint_every_n

6. System information

  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
  • Mobile device name if the issue happens on a mobile device:
  • TensorFlow installed from (source or binary):
  • TensorFlow version (use command below):
  • Python version:
  • Bazel version (if compiling from source):
  • GCC/Compiler version (if compiling from source):
  • CUDA/cuDNN version:
  • GPU model and memory:

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.