tensorflow / tensorflow/models

Minimum training step and loss display frequency not documented

Open
#10,578 0 comments 0 reactions 3 assignees View on GitHub

Nobody has claimed this yet.

models:research:odapi type:docs
Dominant language
Python
Stars
77.7k
Forks
44.8k
PR merge metrics
No merged PRs in 30d

Description

Prerequisites

Please answer the following question for yourself before submitting an issue.

  • I checked to make sure that this issue has not been filed already.

1. The entire URL of the documentation with the issue

2. Describe the issue

I tried to run 20 training steps, with a loss calculation every 2 steps with the OD API, using the following command line:

model_main_tf2.py
  --model_dir="/this/directory/"
  --pipeline_config_path="/this/pipeline.config"
  --log_dir="/this/log/directory"
  --num_train_steps=20
  --num_eval_steps=2
  --checkpoint_every_n=2
  --eval_timeout=30
  --verbose=0 

However, the training event file only contains the loss calculated at step 100 (there is nothing in the log directory). I modified the pipeline.config file as following:

[...]
train_config {
  [...]
  sync_replicas: false
  num_steps: 20
  [...]
}
[...]

The result was the same however: 100 training steps with a single loss calculation at step 100 on the training set. This let me suppose that there is a minimum to the number of training step (100 steps?) and that the loss display frequency (100 steps?) cannot be tuned. This should to be clearly stated in the documentation of proto files and model_main_tf2.py.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.