tensorflow / tensorflow/models
Tensorflow OD API - Tensorboard doesn't load new data
@pkulzc is already working on this.
Since Aug 19, 2020.
- Dominant language
- Python
- Stars
- 77.7k
- Forks
- 44.8k
- PR merge metrics
- No merged PRs in 30d
Description
Prerequisites
Please answer the following questions for yourself before submitting an issue.
[Yes ] I am using the latest TensorFlow Model Garden release and TensorFlow 2.
[ Yes] I am reporting the issue to the correct repository. (Model Garden official or research directory)
[ Yes] I checked to make sure that this issue has not been filed already.
1. The entire URL of the file you are using
python3 model_main_tf2.py --pipeline_config_path=/models/tf2/20200814_ssd_resnet50_v1_fpn/pipeline.config --model_dir=/model/20200818_ssd_resnet50_v1_fpn
For monitoring:
tensorboard --logdir=/model/20200818_ssd_resnet50_v1_fpn
2. Describe the bug
I am training on a remote machine. To see the tensorboard on my local computer i use local port forwarding:
ssh -L 6006:localhost:6006 myusername@remotemachine
I did this earlier with TF1.15 and everything worked fine. But now I am training with TF2 and the tensorboard isn't refreshing. That means, that i still see the old data from the TF1.15 training. I am not sure if the port is caching somehow or what i do wrong so that the tensorboard won't show my new training data.
3. Steps to reproduce
Training itself works fine. I can see the loss decreasing in the log.
Tensorboard is the problem.
4. Expected behavior
Tensorboard should display my training process.
5. Additional context
6. System information
Remote Machine:
Linux
TensorFlow installed from pip 2.3.0
Python version: 3.6.9
CUDA 10.1
GPU model: GeForce GTX 1080 Ti
Local Computer:
Windows
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.