tensorflow / tensorflow/models

Model stopped to do inference after some steps

Open
#9,569 0 comments 0 reactions 3 assignees View on GitHub

@pkulzc is already working on this.

Since Dec 17, 2020.

models:research:odapi type:support
Dominant language
Python
Stars
77.7k
Forks
44.8k
PR merge metrics
No merged PRs in 30d

Description

I'm using SSD MobileNet v2 Open Images v4 (only works/train in tf1) then I created a conda env with tf1 to train

Setup to train

  • Tensorflow-gpu 1.15
  • Cuda 10.0
  • Cudnn 7.6.0
  • Python 3.7.9

To make inferences a use other conda env, with tf2

Setup to inference

  • Tensorflow 2.4.0
  • Cuda 11.1
  • Cudnn 8.0.4
  • Python 3.8.5

I have two env because some models only works in tf1 and other only in tf2, I use both to trains their respectively models, but I only use tf2 env to make inference, always worked in that way

The model was trained until 300k steps, I exported and model worked well, after some days, I got back to train and with 700k, I exported again and the model practically dont make any inference, in tensorboard the model was improving normally (total loss going down), but have a weird gap in loss graphics
Captura de tela de 2020-12-17 09-57-19

Anyone have some suggestions or same issue?

I tried export the model version with 300k steps again to check if could be some setup problem but the model worked

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.