tensorflow / tensorflow/models

Tensorflow OD API - Tensorboard doesn't load new data

Open
#9,123 2 comments 0 reactions 3 assignees View on GitHub

@pkulzc is already working on this.

Since Aug 19, 2020.

models:research:odapi type:support
Dominant language
Python
Stars
77.7k
Forks
44.8k
PR merge metrics
No merged PRs in 30d

Description

Prerequisites
Please answer the following questions for yourself before submitting an issue.

[Yes ] I am using the latest TensorFlow Model Garden release and TensorFlow 2.
[ Yes] I am reporting the issue to the correct repository. (Model Garden official or research directory)
[ Yes] I checked to make sure that this issue has not been filed already.

1. The entire URL of the file you are using

python3 model_main_tf2.py --pipeline_config_path=/models/tf2/20200814_ssd_resnet50_v1_fpn/pipeline.config --model_dir=/model/20200818_ssd_resnet50_v1_fpn

For monitoring:
tensorboard --logdir=/model/20200818_ssd_resnet50_v1_fpn

2. Describe the bug

I am training on a remote machine. To see the tensorboard on my local computer i use local port forwarding:
ssh -L 6006:localhost:6006 myusername@remotemachine

I did this earlier with TF1.15 and everything worked fine. But now I am training with TF2 and the tensorboard isn't refreshing. That means, that i still see the old data from the TF1.15 training. I am not sure if the port is caching somehow or what i do wrong so that the tensorboard won't show my new training data.

3. Steps to reproduce

Training itself works fine. I can see the loss decreasing in the log.

Tensorboard is the problem.

4. Expected behavior

Tensorboard should display my training process.

5. Additional context
6. System information

Remote Machine:
Linux
TensorFlow installed from pip 2.3.0
Python version: 3.6.9
CUDA 10.1
GPU model: GeForce GTX 1080 Ti

Local Computer:
Windows

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.