tensorflow / tensorflow/models
Model stopped to do inference after some steps
@pkulzc is already working on this.
Since Dec 17, 2020.
- Dominant language
- Python
- Stars
- 77.7k
- Forks
- 44.8k
- PR merge metrics
- No merged PRs in 30d
Description
I'm using SSD MobileNet v2 Open Images v4 (only works/train in tf1) then I created a conda env with tf1 to train
Setup to train
- Tensorflow-gpu 1.15
- Cuda 10.0
- Cudnn 7.6.0
- Python 3.7.9
To make inferences a use other conda env, with tf2
Setup to inference
- Tensorflow 2.4.0
- Cuda 11.1
- Cudnn 8.0.4
- Python 3.8.5
I have two env because some models only works in tf1 and other only in tf2, I use both to trains their respectively models, but I only use tf2 env to make inference, always worked in that way
The model was trained until 300k steps, I exported and model worked well, after some days, I got back to train and with 700k, I exported again and the model practically dont make any inference, in tensorboard the model was improving normally (total loss going down), but have a weird gap in loss graphics

Anyone have some suggestions or same issue?
I tried export the model version with 300k steps again to check if could be some setup problem but the model worked
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.